On June 10 we audited our own workflow engine, turned every finding into a ticket, and put a fleet of coding agents on the queue. By the end of the day, 22 fixes were merged and deployed to production.
Then we ran a second pass designed to prove the first one wrong — and it found 18 more problems, several of them in the fixes the fleet had just shipped. Those got fixed and deployed the same evening.
This post is the honest version of that day: what the agents did well, where they were confidently wrong, and why the second pass is the part I'd tell you to copy.
The setup
The backlog wasn't synthetic. The audit covered the engine that runs our own
product — cancellation propagation, crash recovery, secret redaction in logs,
optimistic locking, output size caps, error-edge routing. Real bugs of the
unglamorous kind: the cancel signal that didn't reach child runs, the
redaction that worked until a registry restart, the {{ variable }} that
slipped through unresolved if you typed it with spaces.
Each finding became a ticket with acceptance criteria, the modules involved, and a verification command. That's the unglamorous part nobody wants to hear: the fleet is only as good as the tickets. An agent picking up "cancel doesn't propagate to subflows, here's the module, here's the failing scenario" ships a fix. An agent picking up "cancellation is flaky" ships a guess.
The loop
Every ticket ran through the same path: agent picks up the ticket, plans against the codebase, implements in an isolated worktree, runs the full test gate, and hands back a branch with a structured report. A human — me — read the diffs and held the merge button. All 22 merges that day were mine; none were rubber stamps, and a few went back with notes.
Throughput wasn't the surprising part. The surprising part was the shape of the failures.
What the second pass caught
After the 22 fixes were deployed, we ran an adversarial review: a fresh set of agents prompted not to fix anything, but to refute — find the cases where the day's fixes were incomplete or wrong. It caught 18 problems. A few examples:
- The new secret redaction was fail-open in one path: restart the node registry at the wrong moment and secrets logged in plaintext again. The fix's tests passed because the tests never restarted the registry.
- The unresolved-template gate caught
{{var}}but not{{ var }}. - A crash between "cancel requested" and "cleanup finished" left runs in a state the recovery process didn't know how to claim.
None of these were caught by the first agents' tests, because the first agents wrote tests that confirmed their own understanding. The refuting agents had no investment in the fixes being right. That asymmetry — builder and skeptic as different processes with different incentives — did more for correctness than any prompt engineering we tried.
What I actually believe after this
Agents need a process, not a prompt. The same models, pointed at the same codebase, produce wildly different outcomes depending on whether the work arrives as a structured ticket through a pipeline with gates, or as a chat message. The process is most of the product.
The merge button has to stay human — not as a compliance checkbox, but because reading 40 diffs in a day teaches you exactly where your agents are weak, and that's the feedback loop that improves the tickets.
Verification is a workload, not a step. Budget as much fleet time for refuting as for building. The 18-fix second pass cost roughly as much as the 22-fix first pass and was worth more per token.
The numbers from that day — 22 shipped, 18 more from the adversarial pass, one human on the merge button — are from the production workspace where we build this product. Next post: what a day like this costs in tokens, and where the spend actually goes.