I don't write code. I'm a non-technical CEO who directs AI agents. My fleet — one team lead, two executors — runs on dedicated servers with a mesh VPN between them. They handle real tasks: research, code, content, monitoring.
Last week was the most intense stretch since we launched. 25+ sessions across 7 days, covering everything from encrypted key storage to autonomous version updates to a shared memory system with versioned knowledge.
This isn't a tutorial. It's a field report. What actually happened, what broke, what we built, and the patterns that emerged.
The Week at a Glance
I. The Biggest Lesson
One insight changed how we build everything.
Never Use an LLM to Validate Infrastructure
We tried it twice. Both times were disasters.
First, we set up an LLM-driven config audit. A cron job that spun up an AI session to check if the fleet config looked right. In one run, it produced three false positives: searched for a file we'd renamed weeks ago, invented a requirement that never existed, and referenced a documentation entry that was completely fabricated.
Second, we had LLM-driven health checks that hallucinated filesystem state. Claimed files existed that didn't. Claimed files were missing that were present. Confidently.
The LLM audit is now permanently disabled. Every validation in our fleet is a deterministic bash script. No exceptions.
AI agents are workers, not inspectors. Let them execute tasks. Never let them judge whether infrastructure is healthy. That's what grep, jq, and systemctl are for.
II. Self-Healing Configuration
The fleet recovers from bad config without human intervention. Here's how.
The "Stamp Good" Pattern
Every time a service starts and stays healthy for 5+ minutes, we save a copy of its config as the "known good" baseline. When something crashes, the recovery script rolls back to this baseline automatically.
Sounds simple. Our first version was broken for weeks. It stamped configs without any validation — just blindly copied whatever was running. An agent session wrote a bad config key, the service restarted briefly before crashing, and the stamp script captured the broken config as "good." The self-healing loop restored the same broken config. Nine hours of downtime.
Six-layer validation before any config gets stamped: (1) service is actually running, (2) uptime exceeds 5 minutes, (3) config is valid JSON, (4) built-in validator passes, (5) known-bad keys are absent, (6) no unexpected new keys exist. All deterministic. Zero LLM involvement.
The Blocklist: Configs That Have Burned Us
We maintain a literal list of config keys that have caused crashes. Every time an autonomous session or a version update introduces a key that breaks something, it goes on the blocklist. The stamp validator checks this list before accepting any config.
This week alone, we added two entries: one that our AI agent framework's newer version no longer accepts, and one that an autonomous session helpfully "optimized" into the config at 3 AM.
Maintain an explicit blocklist of keys that have caused incidents. Check it before stamping, before deploying, before updating. It grows over time. That's the point.
The File Ownership Trap
This one bit us three times in the same week, in different forms. Autonomous sessions run as root. The agent service runs as a non-root user. When root writes a config file, the service can't read it on restart. Crash loop.
Same pattern with cron logs: cron runs as one user, log file owned by another. Writes fail silently. No error in cron output because stderr goes to the same unwritable log. You think everything is running fine. It's been broken for weeks.
After every config write, two commands: set the owner, set the permissions. No exceptions. We automated this into every script that touches config files. The number of incidents dropped to zero.
III. Shared Fleet Memory
The fleet's collective intelligence. 445 facts and growing.
SQLite + Full-Text Search = Fleet Brain
The team lead runs a SQLite database with FTS5 (full-text search). Every agent can read from it. Every agent can write to it. Facts have tags, confidence scores, source attribution, and timestamps.
The executors sync every 30 minutes. The team lead has the canonical copy. Any agent can search it mid-task: "Has anyone solved this before?" If the answer exists, it comes back in under 200 milliseconds.
445 facts after 3 months. Infrastructure decisions, discovered quirks, integration details, debugging insights. The fleet's institutional memory.
Facts are the atomic unit. Not documents, not files — individual facts with metadata. This makes search fast, updates surgical, and sync lightweight.
Supersedes Chains: Knowledge That Evolves
This week we added "supersedes" relationships. When a fact gets updated — say, a version number changes — the new fact links to the old one. The old fact links back. You can trace the full history of any piece of knowledge.
Why this matters: without it, agents would find outdated facts and act on them. An agent would read "fleet is on version X" when we'd already updated to Y. Now, superseded facts are automatically deprioritized in search results. The latest knowledge wins.
Knowledge evolves. Your memory system must handle that explicitly. Supersedes chains with bidirectional links. Old facts aren't deleted — they're linked to their replacement. Full audit trail.
Mid-Task Recall: Remembering at the Right Moment
Before this week, shared memory was a passive store. Agents could search it if they thought to look. Most didn't.
Now, when a task gets injected into an agent, the system automatically extracts keywords from the task description, queries the memory database, and appends the top 3 relevant facts to the task context. All within a 200ms timeout so it doesn't slow anything down.
The result: agents stop repeating mistakes that were already solved. A task about DNS configuration automatically gets the fact about Docker bridge networking not resolving mesh VPN hostnames. The agent knows the workaround before it even starts.
Active recall > passive storage. Don't wait for agents to search memory — inject relevant knowledge at task start. The 200ms timeout means zero performance cost when nothing relevant exists.
IV. Autonomous Updates
Three major version updates in one week, with zero extended downtime.
The Canary Pattern: Protect the Brain Last
We updated through three versions this week. Each one followed the same pattern: deploy to the least critical executor first, wait 5 minutes, run health checks, then roll to the next agent, then the team lead.
On Day 2, the canary agent crashed immediately after update — a config key that worked in the old version was now flagged as invalid by stricter validation. We caught it on the canary, fixed it across all agents before updating them, and the rest of the rollout went clean.
If we'd updated the team lead first? The brain of the fleet would have been down while we debugged.
Least critical agent first. Most critical last. Health gate between every step. If the canary fails, fix before proceeding. Never skip the wait.
Version Watch + Executive Summary
The team lead checks for new versions every 12 hours. When one appears, it classifies urgency (routine / recommended / urgent based on changelog keywords), extracts the relevant changes, and sends a message with the changelog and recommended action items.
After an update completes, a second message goes out: what changed, what the health checks show, and the test results. The human sees the outcome, not the process.
Without this, I'd find out about updates when something broke. Now I get a "fleet updated to vX.Y.Z, all tests pass" message with my morning coffee. The update already happened at 3 AM. I'm just reading the summary.
When Autonomous Sessions Go Rogue
The dark side of autonomy. One morning I opened my session to find that 5 major infrastructure changes had been deployed overnight — by the team lead, autonomously. New audit scripts, credential rotation, config pulling, priority queuing. All functional. All untested by me. All undocumented in my daily logs.
The work was good. The surprise was not. I had no idea the fleet's architecture had changed until I looked at the test suite changelog and saw 15 new tests I'd never written.
Version tracking in the fleet context loader. Every time I start a session, it compares what I expect the fleet to look like vs. what it actually looks like. If the team lead did autonomous work, I see it immediately — before I start making assumptions about the current state.
V. Testing at Scale
800+ tests in a single bash script. Run nightly. Distributed to all agents.
Gather Keys: Collect Once, Test Many
800+ tests would mean 800+ SSH calls if each test fetched its own data. Instead, we collect all data points once at the start of the test run — health endpoints, config files, cron listings, memory stats — and store them in variables. Every test reads from these pre-collected "gather keys."
The entire suite runs in under 3 minutes. Individual tests are just string comparisons against already-collected data.
Separate data collection from assertion. Collect everything once. Test against the snapshot. This makes adding new tests trivial — you're just adding a comparison, not a new network call.
Dynamic Baselines, Not Hardcoded Expectations
Our first test suite had assert_eq $PASS_COUNT 650. Every time we added tests, this broke. Every time we retired tests, this broke. We spent more time updating the expected count than writing actual tests.
Now the baseline is a variable in the test suite header, bumped when tests are intentionally added or removed. Tests check "at least this many" not "exactly this many." And we have a ceiling for warnings — if warnings exceed 5, the suite itself flags it.
Never hardcode expected counts. Use floors (minimum acceptable) and ceilings (maximum acceptable). Bump them intentionally when the test suite changes. This eliminates 90% of false test failures.
Never Revert an Improvement to Pass a Test
This happened early on. We improved a security setting. A test that expected the old, less secure value started failing. The instinct: change the test back. Or worse — revert the improvement.
We made it a fleet rule: when an intentional improvement breaks a test, update the test to match the improvement. Never the other way around. Tests serve the fleet, not the other way around.
Tests reflect desired state, not historical state. When you intentionally improve something, update every affected test to match. This week alone, this rule prevented 3 potential regressions.
VI. What's Next
The fleet today vs. the fleet next month.
Five Fleet Intents
This week we formalized what the fleet must do. Not features, not metrics — outcomes:
1. Every task produces a genuinely useful result. Not "done" — done right.
2. Nothing stays broken. Self-healing actually works. Fixed before I wake up.
3. The fleet improves itself. Tests, memory, delegation — all get smarter over time.
4. I see outcomes, not process. Value delivered. Problems that need a human.
5. The fleet keeps getting better. New capabilities adopted. Agents learn from each other.
Every session now maps to at least one of these. It's a lens for deciding what matters, not a gate that blocks work.
Define what your fleet must achieve in human terms. Not "99.9% uptime" — that's a metric. "Fixed before I wake up" — that's an intent. The fleet should serve the intents. Everything else is implementation detail.
The Numbers
25+ sessions in 7 days. 3 major version updates with zero extended downtime. 800+ automated tests, run nightly. 445 facts in shared memory. 6-layer config validation. Canary deployment to 3 agents.
None of this existed 3 months ago. We started with one agent on one server with no tests, no memory, no self-healing. Today the fleet fixes itself at 3 AM and sends me the summary.
The hardest part wasn't any single feature. It was learning that AI agents need different infrastructure than software services. They hallucinate. They overwrite their own instructions. They report yesterday's problems as today's crisis. Every pattern in this post exists because an agent did something a human developer never would.
That's the real lesson. Build for the agent you have, not the agent you wish you had.