Engineering practice
Four times my own verification lied to me in five days
Everything I shipped in the last five days was verified before I called it done. Four of those verifications were wrong, in the way that is hardest to catch: the check reports success in exactly the dimension you are watching while being wrong in a dimension you are not.
None was caught by an alarm. All four were caught by reading something I nearly skipped.
1. The cache hit that made a live deploy look like a no-op
I uploaded a small mu-plugin to the Practically Natty storefront. One file, front-end only, that emits an Organization node and a reference to my person entity in the page head. Before it, the storefront served about 585 KB of HTML and zero application/ld+json blocks — no machine-readable statement of what the company even is.
After the upload I curled the homepage. HTTP 200. No schema in the output. And the response was byte-identical to the capture I had taken before the deploy: 585,279 bytes, to the digit.
Byte-identical is the tell. An upload that failed, a plugin that fatals, a hook that never fires — none of those produce byte-identical output. Byte-identical means you are not looking at a new render at all. The header said so plainly: x-litespeed-cache: hit.
A cache-busting query string returned the same page with the markup in it. It had been there the whole time. I purged, re-checked the clean URL, and confirmed shop and cart still returned 200. As of this writing that plain URL serves 585,884 bytes with one JSON-LD block — a 527-byte block that the cached copy had told me was not there.
The lesson is narrower than "caches exist." An unchanged byte count is evidence about the cache, not about the deploy, and I had been reading it as the second thing. On a cached site: cache-buster first, then purge, then re-verify the clean URL. Three steps, in that order, or you learn nothing.
2. The list that agreed with me
I wrote up the bigger version of this one in the post about the 25 fabricated reply threads. The small trap underneath it is the part worth repeating.
The drafts endpoint on my outreach engine returned 20 rows. The queue actually held 47. The endpoint has an undocumented default page size, and the number I had been quoting myself for days — "46 pending" — was a capped list plus an assumption, treated as a measurement.
It never errored. It returned rows, with a 200, in the shape I expected. A capped list does not disagree with you; it stops early and lets you finish the sentence yourself. The complete view lived on a different endpoint, and it held four internal test drafts I did not know were queued.
If a list endpoint can be capped, its length is not a count. Ask for the total explicitly, or read from the view that is defined as complete.
3. The confirm dialog that named a different URL than the page
Submitting new URLs in Bing Webmaster Tools. I inspected one URL, then inspected a second one in the same page session. The input field showed the new URL. The results panel showed the new URL. The page heading showed the new URL.
The confirmation dialog — "Are you sure you want to submit following URL" — named the previous one.
The likely mechanism: the in-page inspection updated the panel without updating the urlToInspect query parameter, and the submit action read the parameter, not the panel. Every surface I was watching had moved. The one that mattered had not.
I cancelled instead of submitting, redid it by navigating with an explicit urlToInspect parameter, and used the bulk URL Submission textarea for the rest — it renders every URL as plain text before you submit, which is exactly the property you want.
The confirmation surface is the last place the real payload appears in plain text before it leaves. Read it. If it disagrees with the page you think you are on, the page is the thing that is wrong, and cancel is free.
4. The check that could not go red
The oldest of the four, and the purest.
Four checks across three CI and deploy scripts enforced a rule that a specific configuration value must never appear in the built bundle. Each did it the obvious way: grep the bundle for that value's literal string, fail if found. Cheap, sensible, green.
Then the value was changed. All four became scans for a string that existed nowhere. They kept passing. They could not do anything else. The control was neutralised by the exact maintenance event it existed to survive.
The repair was to stop relying on the sample and add the invariant alongside it: the bundle must never contain the config key names, since that is what any accidental serialisation drags along regardless of the values. A guard you have to remember to edit every time the thing it guards changes will eventually not be edited.
The general version, which I now apply to every invariant check I write: negative-test it by injecting the violation and watching it go red. When I added entity guards to three repositories this week, every one was proven that way — inject a second person id, a givenName on what should be a bare reference, a duplicate organization node, and confirm the suite goes red — before I trusted a single green run. A guard that has never failed is a guard you have not tested.
The shape
Put the four next to each other and they are the same failure:
| What I watched | What it reported | What was actually true |
|---|---|---|
| Homepage HTML after a deploy | 200, unchanged bytes | I was reading a cached copy |
| Draft queue length | 20 rows, no error | 47 rows existed |
| The URL on screen | The new URL, everywhere | The submit path held the old one |
A grep guard in CI | Pass | It could no longer fail |
Every one succeeded in the dimension under observation. The status code was honest. The row count was honest. The rendered page was honest. The exit code was honest. What was being asserted had quietly stopped being what I cared about, and nothing in the system can notice that — from inside the check there is no difference between "clean" and "cannot see."
The rule I keep relearning: an instrument must prove it can see a one before you trust its zero.
Why this gets worse with agents
I run a lot of my work through agents now, and this is where it bites. Agent output fails in a specific, uncomfortable way: it is well-formed. It parses, returns the right shape, exits zero. It just asserts something about the world that is not true. The reply thread that never existed. The count that was a cap.
So the verification step is where all the attention goes — and a verification that lies is worse than none, because it is the thing that makes you stop looking.
I now apply the same suspicion to my own auditing. This week I ran an entity audit across six of my domains with roughly thirty agents: one evidence-gathering auditor per property, each required to quote verbatim, then up to four adversarial verifiers per property whose default position was "this finding is wrong, refute it." About 1.48 million subagent tokens. Of 24 findings proposed, 14 survived and 10 were killed — including two of my own suspicions, which the verifiers were right to throw out.
Ten out of twenty-four. Shipping the unverified list would have "fixed" ten findings the evidence did not support, and every one of those fixes would have come with a green check. One of the ten I made anyway — but on a different argument than the one it was born with, which is a decision you make with your eyes open, not a defect a check caught.
The through-line is not sophisticated: know which dimension your check actually reads, and make sure it is the dimension you care about. Then feed it a known positive and watch it fire. Everything else — the status code, the byte count, the row count, the exit code — is the machine telling you it is running. It is not telling you it is right.
Clayton Camera
Technical founder and software engineer in Orlando, Florida. Builds AI systems, autonomous agents, and full-stack software — including OrbitRoute and KnockFiber. Client work at workwithclayton.com.