Systemd, macOS LaunchAgent & Docker Healthcheck Deadlock Watchdog Generator
In standalone Docker Compose, restart: unless-stopped only fires when PID 1 exits -- a deadlocked Node.js event loop, wedged CUDA/Metal runner, or hung socket stays stuck in Up (unhealthy) forever while users hit HTTP 504. Generate coordinated Docker Compose healthchecks, Linux systemd WatchdogSec + Docker Events auto-heal units, macOS LaunchAgent plists, and SIGSTOP chaos verification CLIs.
1. Service, Probe & Supervisor Policy
2. Coordinated 4-Layer Watchdog Policies
docker-compose.ymlLayer 1 · Docker Compose Healthcheck & Cgroup Guard
# Generated Docker Compose Healthcheck + Memory Guard
Why restart: unless-stopped Misses Deadlocked Containers
Docker's built-in restart policy watches kernel process exit codes, not L7 responsiveness. Combining container healthchecks with host-level systemd or launchd supervision closes all four failure modes:
| Failure Mode | PID 1 State | restart: unless-stopped Alone |
Coordinated 4-Layer Watchdog Outcome |
|---|---|---|---|
| Clean Crash / Uncaught Exception | Exited (1) |
Restarts with exponential backoff | Restarts + logs structured unit failure count in journald / launchd |
| Cgroup Memory Leak Spike | Exited (137) OOMKilled |
Restarts blindly without heap cap | MemoryHigh throttles reclaim first; mem_limit + heap flag bounds blast radius |
| Event-Loop / GPU / Pool Deadlock | Up (unhealthy) -- PID 1 alive! |
Does NOTHING (stays wedged forever) | Detected in interval * retries; auto-healer issues bounded docker restart |
Alpine localhost -> ::1 Trap |
False unhealthy loop |
Marks healthy app unhealthy |
Explicit 127.0.0.1 IPv4 + image-native probe (node / pg_isready / wget) |
Companion Diagnostic Utilities & Docker Playbooks
Reproducing missing curl binaries, IPv6 ::1 connection-refused probes, and database cold-start race conditions.
Distinguish kernel cgroup OOM kills from 10-second stop_grace_period SIGKILL timeouts using docker inspect.
Paste multi-line docker inspect, journalctl, and nginx 502/504 logs locally to isolate cascading failures.
Convert raw docker run commands into UFW-safe 127.0.0.1 Compose manifests with log rotation and memory limits.
Frequently Asked Questions
Why does Docker Compose leave an unhealthy container running even with restart: always or unless-stopped?
In standalone Docker Engine (outside Swarm or Kubernetes), container restart policies monitor kernel process termination, not healthcheck status transitions. When a Node.js, Python, or GPU worker deadlocks while PID 1 remains alive, Docker updates State.Health.Status to unhealthy and emits a health_status: unhealthy daemon event, requiring a lightweight host systemd or launchd remediation unit to trigger a clean restart.
Why should container healthchecks target 127.0.0.1 instead of localhost?
Inside Alpine and Debian containers, localhost often resolves to IPv6 ::1 first. If your upstream service binds exclusively to IPv4 0.0.0.0 or 127.0.0.1, wget http://localhost:PORT fails immediately with Connection refused and falsely marks a healthy service as unhealthy.
How can I safely test my deadlock watchdog without crashing production data?
On a staging container, send docker kill --signal=SIGSTOP <container> to freeze PID 1's threads in-place while keeping the process alive. Watch docker inspect --format '{{.State.Health.Status}}' transition from healthy to unhealthy after interval * retries seconds and confirm your auto-heal unit executes a bounded graceful restart.