Your tests pass. That's the problem.
An AI agent writes the implementation, the specification, the tests, and the test fixtures. Then it demonstrates that the implementation satisfies its own definition of correctness. That is not verification; that is the agent grading its own homework. A green checkmark from a suite the agent wrote proves the code agrees with itself, and it is the evidence almost everyone ships on.
Verify against the promises, then try to make it fail.
1. Write the invariants from the claims, not from the code. Every system makes falsifiable promises: "never lose data," "debits equal credits," "a failed batch never advances the checkpoint." List them before you read a line of the implementation. The moment you derive your checks from the code, you inherit the code's blind spots, which are the agent's blind spots.
2. Plant real defects and count what the suite kills. Inject one semantic bug at a time into the paths that carry the promises: an off-by-one on a boundary, a swallowed exception in the write path, a checkpoint that advances before the commit. Run the project's own suite each time. The kill count is the honest measure of what those green tests are worth. Across five engagements, suites caught 5/11, 7/11, 14/16, 11/13, and 14/20 planted defects. All five are public, proof chain included — the most recent a default-install account takeover the maintainer fixed the same cycle, its withheld exploit now shipping as the regression test. Every one of those suites was fully green when we started.
3. Pre-register the attack plan, blind. Hash the plan and anchor it to a clock you don't control before reading the implementation. This is what makes the result evidence rather than storytelling: the findings can't have been reverse-engineered from whatever you happened to stumble on. Every read here ships the hash and the command to check it.
4. Read the escapes; they point at fixtures and wiring, not missing tests. When a planted defect survives, the cause is almost never "no tests." It is a fixture whose numbers divide evenly, so the remainder branch is vacuous, or a fixed bug whose component has an exhaustive unit test while the dispatch that selects the component has none. In our most recent engagement, a 3,632-test suite caught 14 of 20 planted defects, and two of the six escapes were re-breaks of bugs the project had already fixed and documented. The fix existed; the fence didn't.
Three checks that need no vendor.
- Mutation-test your money path. Pick the ten lines where a silent bug costs real money or real data. Break each one deliberately (flip a comparison, swallow an exception, skip a guard) and run your suite. Every mutant that survives is a bug your tests would wave through in production.
- Check whether your fixtures can even see the bug. A reconciler fixture of 70,000 across 7 periods divides evenly, so the leftover cent is always zero and the assertion on it is vacuous. Make the numbers awkward on purpose.
- Test the wiring of your own past fixes. Grep the codebase for issue references in comments; most of them mark a bug you already paid for once. Then check whether a test exercises the call site of the fix, not just the fixed component. If deleting the dispatch line keeps the suite green, the bug is one refactor from coming back.
If the results make you uncomfortable, that discomfort is the finding. The agent that wrote your code cannot supply it.
Want your suite measured every month?
That is what Trenyx does: every month we plant realistic bugs in your code, one at a time, run your own tests against each, and hand you the score, every miss, and the test that would have caught it.