--- title: Heartbeat Monitor Architecture updated: 2026-06-11 ---
The heartbeat monitor checks every local service every 60 seconds via a launchd-triggered shell script. It detects dead ports, unresponsive health endpoints, CLOSE_WAIT socket leaks, and CPU spikes, then auto-restarts the affected service.
Location: `projects/heartbeat-monitor/heartbeat.sh` Launchd label: `com.randy.heartbeat-monitor` Interval: 60s Log: `projects/heartbeat-monitor/heartbeat.log`
| Service | Port | Health URL | Restart Method |
| Portal Auth | 5190 | `/healthz` | `launchctl kickstart` |
| Mission Control | 5173 | `/api/status` | `launchctl kickstart` |
| Document Finder | 8093 | `/health` | `launchctl kickstart` |
| Cost Guard | 7720 | `/health` | `launchctl kickstart` |
| YouTube Synopsis | 5182 | `/` | `launchctl kickstart` |
| Video Analysis | 5191 | `/health` | `nohup node` |
| X Briefing | 5181 | `/` | `launchctl kickstart` |
| Market Research | 8095 | `/mr/status` | `nohup node` |
| Filing System API | 8091 | `/health` | `launchctl kickstart` |
| MUE API | 7710 | `/health` | `launchctl kickstart` |
Root cause: `set -euo pipefail` combined with `lsof` exiting code 1 when a port has no listener. After portal-auth passed, the next service's `lsof` call killed the script with no visible error (logged only to launchd stderr).
Fix: Removed `set -e`. All `lsof` calls wrapped with `|| true` to handle normal "port not in use" exit code.
``` launchd (every 60s) │ └── heartbeat.sh │ ├── lsof :port → check listening ├── curl /health → check responsive ├── check_stuck() → detect CLOSE_WAIT leaks └── check_cpu() → detect CPU spikes │ └── launchctl kickstart service (if unhealthy) ```