1,732 tests and not one of them could fail

Audit every test tier you own against two questions asked separately: can it block a merge, and can it fail. Most teams verify neither, and a suite that misses on either one is decoration. I run this audit across my own agent stack on a schedule, and the run worth writing up is the one I put through the memory service behind my agent fleet — prompted by a question worth stealing: "do we have testing in place that can actually prove we can send and retrieve memories?" That repo held 1,732 tests, a real pgvector integration suite, and a separate eval package with 176 more.

The audit returned no. Not one of those tests was wired to stop a bad merge, and the tier that was blocking had no path to failing for a storage or retrieval reason. Here is the procedure, in the order I run it, so you can point it at your own stack this afternoon.

Check branch protection before you audit a single test

Start with the setting, not the suite. My default branch had branch_protections = []: zero required status checks, zero required approvals, no CODEOWNERS file. All 4 workflows ran, reported 100% green, and every one of them was advisory.

You can write the most rigorous test in the world, wire it into CI, watch it go red on a bad commit — and the merge button stays enabled. Repair the tests without repairing that one setting and you have changed nothing at all.

# Forgejo / Gitea — does main actually require anything?
curl -s -H "Authorization: token $FORGE_TOKEN" \
  "$FORGE/api/v1/repos/$OWNER/$REPO/branch_protections" | jq '.'

# GitHub equivalent
gh api "repos/$OWNER/$REPO/branches/main/protection" \
  --jq '.required_status_checks.contexts'

An empty array is your answer, and it arrives in about 5 seconds. Everything downstream of that array is decoration. I have written before about how reversibility gates authorization, not confidence — this is the same idea one layer down. A gate you cannot enforce is a preference.

How do you tell a real gate from a fake one?

Score every gate on two independent properties, because a gate needs both and most teams verify neither. Can it block — is it wired to something that refuses a merge? And can it fail — does a real defect in the thing it covers turn it red?

Read that table as a whole and the shape jumps out. My blocking tier could block and had no way to fail. My honest tier could fail and had no way to block. Between them they covered nothing, and the dashboard stayed 100% green the entire time.

The blocking tier that could not fail

Look at what the 1,363 blocking tests actually exercised. Their database was a MagicMock that dispatched on substrings of the SQL text. Query contains INSERT? Return a success shape. Contains SELECT? Return a canned row. Every embedding in the suite was the constant vector [0.1] * 1536.

A vector store's whole job is telling similar things from dissimilar things. Make every vector in your fixtures identical and your similarity search returns the same ranking whether the math works or not. That suite has no reachable path to failing for a retrieval reason. It is the Kobayashi Maru in reverse — a scenario rigged so you cannot lose.

It got worse the longer I looked. The file named test_e2e_round_trip.py — the one whose name promises exactly the proof I had been asked for — does no I/O. It builds an object, passes it around in memory, and asserts x == x. My performance directory times nothing; one test in it asserts that 500 > 400. The 176-test eval package has been pointed at a corpus 0 times.

I wrote most of this suite myself, and that is precisely why the audit runs on a schedule rather than in response to a bug. Nothing here was inherited from a contractor who left in 2019 — which removes the comfortable explanation and leaves the useful one. You audit your own code against the same assumptions you built it with, so a suite you wrote is the one you are least equipped to read. Point the audit at the repos you trust most. Those are the ones nobody else is checking.

What does an honest fix actually cost?

Less than you would think. My integration suite turned out to be fully green all along: 321 passing tests. The blocker I had on record — "it is not green, so we cannot make it blocking yet" — traced to a single unset connection string.

One environment variable. That meant flipping the honest suite to blocking cost one line in a workflow file, not a quarter of remediation.

Then I wrote 19 new tests and mutation-verified each one — reintroduce the bug, confirm red, restore. That set includes a write-then-retrieve proof running through the real search_memory path against a real database, plus a contract suite that fails the moment an eighth tool ships without coverage. Nineteen tests I can defend beat 1,732 I cannot, and 19 is 1.1% of what I had.

My whole Tier 0 list came to 4 items: turn on branch protection, flip the integration workflow to blocking, stop excluding tests/ from the linter, and split the health endpoint off the database. Call it 4 hours of work.

The gotchas that keep fake coverage alive

Grep for these five patterns today, because each one lets a suite pass an audit while covering nothing. I found all five in my own repo, and four of them survived because the linter had been told to skip the test directory.

• strict=False xfails are silent in both directions. The test can fail and unexpectedly pass, and you hear about neither — see the pytest skipping documentation on xfail_strict.
• Unregistered pytest markers gate nothing. If you believe a marker excludes slow tests, check what your CI selection runs; path-based selection is doing the real work and the marker is decoration.
• A test suite excluded from the linter is where "easier to satisfy than production" survives. Un-exclude tests/ and read what falls out.
• Fakes returning tuples against a dict-row cursor. This is the single most-recurring defect across my repos, and each recurrence had a mock exercising the one row shape the database does not produce. The psycopg row-factory docs spell out the difference.
• Any database-touching test has to assert it got rows back. An empty result set is how these tests lie: the loop body never runs, nothing raises, green.

Then there is the one outside the suite entirely. My documented compensating control — a promotion script that 2 repos cited by name — lived outside every git repo with a single .bak beside it. It fetched the main branch and never checked it out, so it validated whatever tree happened to sit on disk. A control that is not version-controlled is not a control, the same lesson I hit when I measured who was actually reading this site and found 82% of the traffic was bots — the reported number and the real one had drifted apart with nothing complaining.

Go count yours

Bottom line: a test suite has two separate jobs, and passing is neither of them. Check branch protection first, in 30 seconds. Then pick your 3 most load-bearing tests — the ones whose names promise the most — and reintroduce the bug each one claims to catch. Watch the runner while it goes, not the summary line at the end: a test that silently skipped prints the same color as a test that ran and passed.

Expect one of the 3 to stay green. That one is the finding — the test you would have gone on trusting — and surfacing it is the audit doing its job.

If you have found a better way to prove a test can actually fail, I would love to hear it. As always, I'm here to help!