GEARFLOW LAB // SYSTEMD · LAUNCHD · DOCKER DEADLOCK WATCHDOG 100% Client-Side · Auto-Heal Ready

Systemd, macOS LaunchAgent & Docker Healthcheck Deadlock Watchdog Generator

In standalone Docker Compose, restart: unless-stopped only fires when PID 1 exits -- a deadlocked Node.js event loop, wedged CUDA/Metal runner, or hung socket stays stuck in Up (unhealthy) forever while users hit HTTP 504. Generate coordinated Docker Compose healthchecks, Linux systemd WatchdogSec + Docker Events auto-heal units, macOS LaunchAgent plists, and SIGSTOP chaos verification CLIs.

6 One-Click Production Watchdog Presets: svc=n8n · hostos=systemd · deadlock=1

1. Service, Probe & Supervisor Policy

15s
3x (45s window)
30s
2048 MiB (2.0 GiB)

2. Coordinated 4-Layer Watchdog Policies

docker-compose.yml
Deadlock Detection
45s steady-state
30s boot grace + 3 x 15s
systemd WatchdogSec
60s ceiling
MemoryHigh=1740M / Max=2048M
PID 1 Unhealthy Action
Auto-Restart Armed
120s anti-flap cooldown
Deadlocked PID 1 Auto-Remediation Armed (unhealthy -> Graceful SIGTERM + Restart) Standalone Docker Compose never restarts an unhealthy container whose PID 1 is still alive. Policy #2/#3 closes that gap with a zero-polling event watcher and explicit IPv4 127.0.0.1 health probes.

Layer 1 · Docker Compose Healthcheck & Cgroup Guard

# Generated Docker Compose Healthcheck + Memory Guard
PRODUCTION FAILURE MATRIX // EXITED VS OOMKILLED VS UNHEALTHY DEADLOCK

Why restart: unless-stopped Misses Deadlocked Containers

Docker's built-in restart policy watches kernel process exit codes, not L7 responsiveness. Combining container healthchecks with host-level systemd or launchd supervision closes all four failure modes:

Failure Mode PID 1 State restart: unless-stopped Alone Coordinated 4-Layer Watchdog Outcome
Clean Crash / Uncaught Exception Exited (1) Restarts with exponential backoff Restarts + logs structured unit failure count in journald / launchd
Cgroup Memory Leak Spike Exited (137) OOMKilled Restarts blindly without heap cap MemoryHigh throttles reclaim first; mem_limit + heap flag bounds blast radius
Event-Loop / GPU / Pool Deadlock Up (unhealthy) -- PID 1 alive! Does NOTHING (stays wedged forever) Detected in interval * retries; auto-healer issues bounded docker restart
Alpine localhost -> ::1 Trap False unhealthy loop Marks healthy app unhealthy Explicit 127.0.0.1 IPv4 + image-native probe (node / pg_isready / wget)

Companion Diagnostic Utilities & Docker Playbooks

Docker Compose Healthcheck Failed: curl, ::1 & start_period Fixes →

Reproducing missing curl binaries, IPv6 ::1 connection-refused probes, and database cold-start race conditions.

Docker Exit Code 137: OOMKilled=true vs SIGKILL Triage →

Distinguish kernel cgroup OOM kills from 10-second stop_grace_period SIGKILL timeouts using docker inspect.

DevOps & AI Error Log Decoder (60 Signatures) →

Paste multi-line docker inspect, journalctl, and nginx 502/504 logs locally to isolate cascading failures.

Docker Run to Production Compose Converter →

Convert raw docker run commands into UFW-safe 127.0.0.1 Compose manifests with log rotation and memory limits.

Frequently Asked Questions

Why does Docker Compose leave an unhealthy container running even with restart: always or unless-stopped?

In standalone Docker Engine (outside Swarm or Kubernetes), container restart policies monitor kernel process termination, not healthcheck status transitions. When a Node.js, Python, or GPU worker deadlocks while PID 1 remains alive, Docker updates State.Health.Status to unhealthy and emits a health_status: unhealthy daemon event, requiring a lightweight host systemd or launchd remediation unit to trigger a clean restart.

Why should container healthchecks target 127.0.0.1 instead of localhost?

Inside Alpine and Debian containers, localhost often resolves to IPv6 ::1 first. If your upstream service binds exclusively to IPv4 0.0.0.0 or 127.0.0.1, wget http://localhost:PORT fails immediately with Connection refused and falsely marks a healthy service as unhealthy.

How can I safely test my deadlock watchdog without crashing production data?

On a staging container, send docker kill --signal=SIGSTOP <container> to freeze PID 1's threads in-place while keeping the process alive. Watch docker inspect --format '{{.State.Health.Status}}' transition from healthy to unhealthy after interval * retries seconds and confirm your auto-heal unit executes a bounded graceful restart.