AI agents in production
The dry run that caught 25 fabricated reply threads
I run the outreach for my orbital-compute product myself, with an agent drafting and a human approving. Last night the approved queue was 43 emails. Twenty-five of them were about to arrive in the inboxes of the best prospects on my list with a Re: in front of the subject line — a reply to a conversation that had never happened.
They didn’t send. Not because of a clever guardrail, but because the dry run printed the one field I wouldn’t have thought to look at. Here’s the defect, why it was worse than a bug, and the two habits that caught it.
What the queue actually contained
The engine drafts two kinds of message: a first touch, and a follow-up on an existing thread. Follow-ups get the Re: prefix and quote the prior message, which is correct — it’s a real thread. First touches must not.
The mapping was inverted almost perfectly:
| Bucket | Count |
|---|---|
Re: + never contacted | 25 ← fabricated thread |
Re: + prior contact | 1 |
no Re: + prior contact | 17 — and the subject was byte-identical to the original send, 17 / 17 |
So the genuine follow-ups looked like duplicate first touches, and the genuine first touches looked like replies. The prefix was being derived from the draft template rather than from a lookup against contact history — a distinction that is invisible until the two disagree.
A crash costs you a send. A fake Re: costs you the relationship: the first thing a stranger learns about you is that you’re willing to pretend they already replied. On a list where every name is a company I want to work with for years, that is the most expensive possible failure, and it would have looked like a successful campaign in every metric I track — 43 delivered, deliverability fine, nothing red.
The pagination default that nearly hid it
Before that, a smaller trap. The drafts endpoint returned 20 rows. The queue actually held 47. The endpoint has an undocumented default page size, and the count I’d been quoting myself for days — “46 pending” — was a number I had assembled from a capped list plus an assumption.
The complete view came from a different endpoint entirely. Four of the 47 turned out to be internal test drafts addressed to my own domain, one of them titled “(internal test)”. They would have gone out with the batch.
If a list endpoint can be capped, never treat its length as a count. Ask for the total explicitly, or read from the view that is defined as complete. A capped list doesn’t error — it just quietly agrees with you.
What a dry run has to print
My dry run used to print what most dry runs print: how many messages, to how many recipients, under what cap. All three numbers were correct. All three were useless here.
What caught it was printing, per message, the fields that would embarrass me if they were wrong — the literal rendered subject line next to the recipient’s contact history — and then cross-tabulating the two. The defect was invisible in aggregate and obvious in a 2×2.
A dry run that only reports volume is a progress bar. Make it render the exact artifact the recipient will see, beside the state that artifact claims about them, and group by the combination. Then assert on it: zero messages may claim a prior thread with a contact whose history is empty.
That assertion now runs before dispatch, not as advice in a runbook. The prefix is derived from a contact-history lookup rather than from the template type, and internal domains are excluded from any batch by rule instead of by my noticing them.
The honest result
After the repair, all 43 dispatched: 42 delivered, 1 bounce — a 2.3% bounce rate, down from 20% on the previous wave after a list-hygiene pass. Total outreach to date, 63 emails.
Replies so far: zero. A first-touch null on cold outreach is normal and I’d be lying to say otherwise; the follow-up wave is where the real signal is. I’m publishing the number now, before it’s flattering, because the alternative — quietly waiting until there’s a good one to report — is the exact habit that makes founder metrics worthless.
The transferable part
Agent-drafted communication fails in a specific, uncomfortable way: the output is well-formed. It’s not garbled, it doesn’t throw, it passes every schema check. It just asserts something about the world that isn’t true — a thread that never existed, a prior call, a shared connection. Type systems and status codes don’t see that class of error at all, and neither do dashboards.
The only defense I’ve found that reliably works is to make the dry run show you the claim, next to the evidence for the claim, before anything leaves the building. Volume counts will tell you the machine is running. They will never tell you it’s honest.
Clayton Camera
Technical founder and software engineer in Orlando, Florida. Builds AI systems, autonomous agents, and full-stack software — including OrbitRoute and KnockFiber. Client work at workwithclayton.com.