A pre-registered verification pass on
gr8monk3ys/trading-bot, an
AI-built Python trading system (Alpaca broker, momentum + mean-reversion, backtest
engine, risk manager). 57% of its commits are Co-Authored-By: Claude —
exactly the kind of code Trenyx verifies. Free and unsolicited; the maintainer owes
it nothing.
The suite was green — 1,921 passing. The question is never whether the suite passes; it's whether it would catch a bug in the money path. So I ran the suite as shipped, then injected eleven semantic defects one at a time and measured how many its tests caught.
| result | |
|---|---|
| baseline suite (as shipped) | 1,921 passed · 11 skipped · 0 failed |
| planted defects caught by its tests | 5 / 11 |
| confirmed defects (independent, disclosed) | 8 |
Five of eleven. What escaped its green suite were not exotic: a one-bar lookahead (the backtest peeking at the future), a trailing stop anchored to the entry price instead of the high-water mark, a kill switch that was ignored, a paper→live default that could send real orders, stale marks, and a double-processed final bar. Every one of those ships green today unless someone tests the behavior, not just the code path.
Independent tests — written blind to the target's own suite — then confirmed eight defects in the production path, disclosed to the maintainer first among the ten reproducible issues filed in gr8monk3ys/trading-bot#99 (the safety-adjacent item also routed through their private advisory channel).
CORRECTIONS.md,
along with the practices adopted because of them. A record you can trust is one that
publishes its own misses too. (This first sample's pre-registration rests on commit
order alone; from verify-002 on, the plan's hash is anchored to a public post before
any code is read.)
Open the engagement repo → Read the disclosure
The repo carries the pre-registered attack plan, the baseline, the blind tests, the full planted-defect matrix, the published findings (the advisory-channel item's specifics publish after a fix lands), and a receipt hashing every artifact — apply a patch, run their suite, compare. Denominators are ours and published; these numbers describe this attack, not a universal score.
Every engagement ships the pre-registration, the baseline, the planted-defect matrix, and the findings — disclosed to you first, privately, with the same receipts you see here.