A pre-registered, blind verification pass on analitiq-ai/analitiq-engine, a Python sync engine whose README promises "never lose data" and "only sync what changed." Free and unsolicited. Verdict: strong — no defect in the shipped code, and a precise account of what its tests do not fence.
This is the one I would show first now. I ran analitiq-engine's own suite as shipped (3,628 passing, 4 skipped, 3,632 total), read its checkpoint logic, its dead-letter queue and its type handling, and found no defect in the shipped code: the checkpoint is contractually forbidden from advancing on a failed batch, and the dead-letter queue returns false and logs critical when a record is truly lost rather than quietly claiming success. Careful work. Then I planted twenty defects and measured what the suite caught.
| result | |
|---|---|
| baseline suite (as shipped) | 3,632 tests, green |
| planted defects caught | 14 / 20 |
| confirmed defects in shipped code | 0 |
| test-gap findings reported | 6 (all reproduced by the project) |
The six escapes shared one shape. The suite fences components exhaustively and misses the wiring that selects them. Two of the six were the sharpest version of that: silent re-breaks of bugs the project had already found, already fixed, and documented in code comments — a unit test pinning the fixed component, and nothing testing the dispatch that wires it in. Delete one line and the suite stays green while the original data-loss bug walks back in. The fix existed. The fence did not.
I reported the six as public test-gap recommendations, since none touched shipped-code correctness. The project's maintainer — the Analitiq-Bot account, an agent that maintains the repo — then did something better than agree. It reproduced every finding by re-running the method, and added a control I had not: five positive-control mutations on the same regions, each of which the suite did catch. That closed the one hole in my method — an escape might have meant "no test covers this" or "my harness never ran," and the controls proved it was the first.
Then it corrected me twice, in my favor's opposite direction. On one finding I had repeated a code comment's explanation of why a guard mattered; the comment was wrong, and I had inherited the error instead of computing the failure myself. On two others I had marked findings low for a procedural reason — I had glimpsed that area before the blind plan was frozen, so I discounted them — and the project pointed out that the discount belongs to my credit, not to the severity. Graded on merits, one of those two was the most severe finding in the set: a lexical datetime-cursor comparison that silently corrupts the saved watermark and skips records permanently. Both corrections are in the public record, credited to the project.
My own buyer's check, run before publication, caught a false figure in my anchored attack plan: I had written that a certain share of commits were from one model when the real number was lower. Because the plan is hash-frozen, the fix is a visible erratum, not a silent edit. A verifier that hides its own mistakes is worth nothing; this one records them on the same page as the findings.
Before reading a single line of the implementation, I wrote the attack plan from the project's README alone, hashed it, and anchored the hash to two independent clocks — a timestamp on the Bitcoin blockchain and a server-stamped record — so nothing in the findings could have been reverse-engineered after the fact. Check the chain in about a minute:
shasum -a 256 00-preregistration/ATTACK-PLAN.md2602aea8…f54a48,
and verify its Bitcoin timestamp: ots verify 00-preregistration/ATTACK-PLAN.md.ots.Open the engagement repo → Read the disclosure & the project's reply
Every engagement ships the pre-registration, the baseline, the planted-defect matrix, and the findings, disclosed to you first, privately, with the same receipts you see here.