We run a 3-server AI agent fleet — one team lead, two executors — handling delegation, research, code execution, and monitoring around the clock. Real tasks, real users, real consequences.
Every failure got a number. Every number got a fix. Here are the 20 gotchas that actually matter, ordered by how badly they'll hurt if you don't know about them.
I. The Agent Turned On Itself
The most dramatic failures — when the AI actively works against its own operation.
The Agent Overwrote Its Own Instructions
We asked an agent to "clean up the workspace." It helpfully organized all files — including AGENTS.md, the file containing its operational instructions. Replaced the entire thing with a neat summary. The agent no longer knew how to do its job.
Read-only bind mounts (:ro) for all critical files. Instructions, identity, config — immutable from the agent's perspective.
Same Task Ran 11 Times in an Infinite Loop
Agent finishes a research task. Tries to report results using a hostname. Docker can't resolve it. Callback fails silently. Mailbox retries every 16 minutes. For 3 hours. 11 identical runs.
All callback URLs use raw IPs, never DNS names. Mailbox now has idempotency guards — same task ID can't re-execute if results exist.
Agent Reports Went Nowhere — For Days
Agents "completed" tasks and wrote beautiful summaries... into their own chat session. The orchestrator never received anything. They were typing their results instead of sending them. Chat messages are local.
Made script execution MANDATORY in instructions. Bold. Caps. Repeated three times. Agents still occasionally forget.
Stale Memory = Yesterday's Crisis Reported as Today's
Agent's memory says "Agent2 is down" from yesterday. Agent2 has been fine for 20 hours. But the agent confidently reports a current outage. Memory persists; reality changes.
Rule: always verify via live check before reporting any issue. We call this "Gotcha #136" and reference it in every operational document.
II. Silent Failures That Look Like Success
Everything reports green. Nothing is working. The scariest category.
Ghost Tasks Blocked the Queue for Hours
Tasks complete. Results delivered. But the local inbox still says status: working. Queue full. No new tasks accepted. Agent looks broken but finished everything hours ago.
Report-back must update both local state AND remote callback. Completed tasks move to archive after successful delivery.
One Config Key Crashed the Entire Service
Added capAdd: ["NET_ADMIN"]. Reasonable setting, correct syntax. Service refused to start. Hard crash on any unrecognized key. Agent down for 2 hours.
Verified config schema doc. Check before every edit. Rule: "if unsure whether a key exists, don't add it."
Cron Jobs Ran "Successfully" for 2 Weeks Doing Nothing
Cron runs as openclaw. Log owned by root. Write fails silently. No error in cron mail because stderr goes to the same unwritable log.
Audit every file your cron touches for ownership. If the user can't write it, the cron doesn't schedule.
Health Check Lied: "3 Days Stale" Was Actually Fine
Health endpoint: memory_sync_age_seconds: 284137. Three days stale. SSH in — sync ran 4 minutes ago. The endpoint just calculated the timestamp wrong.
Never trust a single metric. Health endpoints are hints, not truth. Always verify via direct check.
Concurrent JSON Reads Returned Half a File
Two processes read the same task file. One gets {"task": "research comp — truncated. JSON.parse fails. Task vanishes. No crash, no log.
Atomic writes: temp file + rename(). OS 101 that agent frameworks skip.
SIGUSR1 + Disabled Service = Silent Death
Restart=on-failure. SIGUSR1 for reload. Service was manually stopped. Process dies, systemd doesn't restart. No alert. Agent gone.
Check service state before signals. Or Restart=always with the trade-off.
III. Networking Lies
Containers, mesh VPNs, and DNS create a reality distortion field.
Localhost Returned ECONNREFUSED — Service Was Healthy
Service binds to Tailscale IP. Health checks using localhost → ECONNREFUSED. We thought it was down for 3 hours. Running perfectly, just not on 127.0.0.1.
Never hardcode localhost. Resolve the actual interface IP. Burned into every health check script.
Docker Bridge Can't See the Mesh VPN
Agent in Docker calls a peer via Tailscale hostname. DNS fails. Bridge network doesn't know Tailscale exists. All cross-agent communication dead.
network: "host" in Docker config. Plus hardcoded Tailscale IPs. Belt and suspenders.
SSH From Docker — Tool Not Found
Delegation script SSHs to a peer. Works on host. Docker: no openssh-client. Error swallowed. Task silently dropped.
Install tools in Docker setupCommand. Never assume container has what the host has.
Hardcoded Callback Broke Peer-to-Peer
All delegate scripts had the orchestrator's IP as callback. Agent2→Agent1? Results go to orchestrator. Agent2 waits forever. Only hub-spoke worked.
Auto-detect own IP, set callback to self. Results return to whoever sent the task.
IV. Infrastructure Traps
Gotchas that hit before your agents even start working.
"Sandbox Off" Still Runs Docker
sandbox.mode: "off". Expected bare-metal. Still Docker. Every command containerized. Hours debugging "missing" binaries on the host.
Sandbox controls permissions, not Docker. Add explicit bind mounts for everything needed.
4GB Servers OOM on Package Install
pnpm install eats all RAM. Process killed. Cloud VPS ships with zero swap.
First command ever: fallocate -l 4G /swapfile && mkswap && swapon.
chmod 600 on Tokens Broke Docker Auth
Security instinct: chmod 600. Docker runs as different UID. Can't read tokens. Auth fails silently.
644 for Docker-readable tokens. Security theater that breaks functionality isn't security.
140KB CLI Argument Hit Linux's Invisible Limit
Sync script passed 140KB JSON as CLI arg. ARG_MAX: ~128KB. Process dies with zero error. Fleet sync silently broke.
Pipe through stdin or temp files. Never >10KB as CLI args.
V. The AI-Specific Weirdness
Gotchas that only exist because your workers are language models, not scripts.
Typing Indicator Survived Agent Death
Agent starts task. Telegram: "typing..." Session crashes. Indicator stays forever. Users think it's working. Dead for hours. Nobody notices because it looks busy.
Explicit cleanup on session end. Watchdog clears stale indicators after timeout.
Search Broke on Hyphens
gotcha-143 → nothing. gotcha 143 → found. SQLite FTS5 treats hyphens as token separators. Our entire knowledge base uses hyphens.
Spaces not hyphens in queries. A punctuation mark cost hours of debugging "missing" knowledge.
The Meta-Gotcha
The biggest lesson isn't any single failure — it's that AI agents find failure modes humans never would. They'll overwrite their own instructions, report yesterday's problems as today's crisis, and run the same task 11 times in a loop.
Every gotcha above was discovered in production. Not testing, not staging — production. The only defense is documenting everything, verifying before reporting, and assuming that whatever can break will break in a way you didn't expect.
90 days. 89 gotchas. Zero extended outages. Not because nothing breaks — but because every break becomes a numbered lesson the fleet never repeats.