ledgerly: the strongest test suite I've measured.
A pre-registered, blind verification pass on jakethehoffer/ledgerly, a TypeScript engine that turns Stripe webhook events into balanced double-entry journal entries. Free and unsolicited. Verdict: exemplary.
Most test suites are green. The question that matters isn't whether the suite passes. It's whether the suite would catch a money bug introduced tomorrow. So I ran ledgerly's own 810 tests as shipped (all green), then injected sixteen semantic money-defects one at a time and measured how many its suite caught.
| result | |
|---|---|
| baseline suite (as shipped) | 810 / 810 green |
| planted money-bugs caught | 14 / 16 |
| confirmed defects in shipped code | 0 |
| findings disclosed | 1 (low, benign) |
Fourteen of sixteen is the best result of any target I've put through this, and more tellingly, its suite caught the balanced-but-wrong mutations: entries that still balance to the cent (debits = credits) but post to the wrong account. Those are the ones weaker suites wave straight through, because a balance check can't see them. ledgerly's tests assert the specific accounts, not just the arithmetic.
The two defects that escaped both trace to a single localized gap, in the tests, not the shipped code. One reconciler fixture re-spreads its amount over months that divide evenly (70000 over 7 = 10000 each, no remainder), so the branch that places a leftover cent on the last month is never exercised. A regression there would ship green. The fix is one fixture whose numbers don't divide evenly. I filed it, gratefully, as a public feature request: ledgerly#2. There is no security or money bug in the shipped code.
The plan for this read was written from the project's README alone, hashed, and timestamped to the Bitcoin blockchain before the code was cloned. The plan stays private while the fault catalogue it draws on does. The findings, the report and the maintainer's response are the receipts on this page.
Engagement repo: private Read the disclosure
Want your codebase put through the same discipline?
Every engagement ships the pre-registration, the baseline, the planted-defect matrix, and the findings, disclosed to you first, privately, with the same receipts you see here.