More than half the bugs your tests miss are in code they already run

Over the last few weeks I planted 1,598 realistic bugs, one at a time, into private copies of 100 public codebases, and ran each project's own tests against every one. 1,591 of them produced a usable result, and the tests caught 1,080, about two in three.

The number I keep coming back to is where the other 511 sat. Of the misses I have coverage data for, more than half were in lines the tests ran in full.

How a run works

Each run takes one slice of a codebase (a package, a crate or a module) and the tests that cover it. An AI model writes about 16 bugs for that slice, working from a copy of the source with the test files removed. Some bugs follow a catalogue of bug shapes built up from earlier runs and earlier code reviews; the rest are written for that particular codebase. The list is committed before the runs start, and in none of these runs was a planted change edited after its results came in.

Each bug then gets its own CI run on a private copy, alone. If any test fails, the bug counts as caught. Before anything counts, the unmodified code is run at least three times, and any test that already fails there is set aside. Catches are run a second time to check that they repeat (all but 11 of the 1,080 were). Three catches that didn't repeat were thrown out, along with three bugs that made the suite hang and one that broke the build. A coverage run on the unmodified code records which lines the tests execute. Nothing touches the projects' own repositories or CI.

Read the numbers with this in mind

The 100 codebases are not a random sample. Most are products from funded companies, picked as teams I might sell to, and a few are from larger companies. Most are open source; some are under source-available licences. In each, the slice was chosen where a real test suite exists, which favours well-tested code.

Each run covers one slice with the tests that cover it. A team's other suites might catch some of the misses, and if a team shows that one does, its score changes.

In five of the runs, my setup left out a test dependency the project's own CI installs, so some of its tests never ran and were set aside as already failing. At most one miss is affected.

Planted bugs are not real bugs. They are written to look like mistakes a developer could plausibly ship, but a model wrote them, and nothing yet checks that every one of them could be caught by some test. So far, only seven of the misses have been shown to be catchable by a written test.

Removing test files doesn't hide everything. In some runs the writer could still see test names, or comments about what a test checks, in the code it read. A later check found more of these than I had recorded at the time.

The per-kind numbers further down rest on 20 to 108 bugs each. Treat them as leads.

The scores

The median run caught 69%, and the middle half of the runs fell between 56% and 81%. None caught all 16; the best caught 15. Sixty-eight of the 100 missed a quarter or more, and 12 missed more than half.

Share of planted bugs each codebase's own tests caught

100 codebases · one dot each · no names

Each dot is one codebase: the share of its planted bugs that its own test suite caught. The shaded band is the middle half of the runs; the dashed line is the median.

Show the scores as a list

Where the misses were

511 bugs got past. For 472 of them, the coverage run shows whether the planted lines ran:

256 (54%) were in lines the tests ran in full.
166 (35%) were in lines no test ran.
50 (11%) were in lines the tests ran only in part. In many of these the changed code itself never ran, so I count them in neither group.

The other 39 have no usable coverage data: some runs had no coverage step, and some lines don't map to anything a coverage tool reports.

Every one of those 256 planted lines would show as covered in a line-coverage report. Coverage tells you a line ran. It doesn't tell you anything checked what the line did.

The never-run third is the familiar problem, and coverage already points at it. The other half is the one coverage hides.

Which bugs slip

Among the 16 kinds with at least 20 plants, counted under the label each bug had when it was written (two labels for one shape, a safe default flipped to an unsafe one, are counted together), ranked by how often they slipped:

Slipped most:

A name or path prefix matched without its separator, so a sibling like main2/ counts as main: 18 of 24 slipped.
An error swallowed and returned as a success: 23 of 43.
A lookup key that loses one of its fields, so two records that differ only in that field match, such as a price in another currency: 29 of 62.
A timestamp read in seconds where it was written in milliseconds, or the other way round: 15 of 35.
A validating pattern that loses its start or end anchor, so it accepts input it should reject: 13 of 31.

Slipped least:

A function that returns the fallback instead of the value it was meant to keep: 8 of 39.
Two conditions of an access check joined with OR where both must hold: 8 of 34.
Two conditions of an ordinary check joined with AND where either one should do: 25 of 93.

Can't my coding agent just do this?

Partly, yes. I gave five fresh coding agents a plain request on five projects: plant 16 bugs, run the tests, say which ones were caught. They counted well: on 79 of 80 bugs their verdict matched a mechanical re-run. They left the code clean.

What they didn't give me was a measurement. Every agent read the tests before choosing its bugs and aimed at the gaps it suspected, and one swapped a missed bug for another after seeing the result. In four of the five projects the agent's score came out below the score from bugs written blind and locked in advance, by as much as 31 points. That's useful for finding a gap once. It isn't a number you can compare next month or with another project.

What to do about it

Assert the value, not the call. If a function writes a record, read it back and check the field that matters.
Test the failure path on purpose. Make the dependency fail and assert that the caller sees an error, not an empty success.
Test the value that sits on the boundary: the limit itself, the cutoff date, the name one character from too long.
Plant one bug yourself. Pick a function you trust your tests on, break it in a small, realistic way, and run the suite. If nothing fails, you have learned more than a coverage report will tell you.

If you'd like a run on your own code, email sk@trenyx.io. The first report is free.

Disclosure: I run Trenyx, which sells these runs as a monthly service. The bugs are written by an AI model; the CI runs, the rechecks and the counting are mechanical. Runs from before the step that removes test files are not included, and one run was left out because none of its runs on the unmodified code finished. The analysis behind these numbers was frozen before this was written, the totals were recounted independently, and every verdict was recomputed from the cached CI output. These numbers cover 100 of the first 101 runs; the running total on trenyx.io includes the runs since, so its count is larger. Alongside these runs I do AI-assisted security reviews of open-source code; eight of those reports are now published advisories, three of them rated high. I wrote this with help from Claude on editing and on checking every number against the frozen analysis.