TrenyxStart
Menu
Sample report · a real run on an open-source project; its name, file names and identifiers are changed, the results are not
Planted-defect run · an open-source Stripe billing library (TypeScript) · 25 September 2026

Your tests caught 4 of 16 realistic bugs.

We planted 16 bugs a developer or an AI assistant could plausibly write, one at a time, and ran your own test suite against each. 12 got through. Three sit in code no test runs at all; three sit in code your tests run but never check. Every miss below comes with the test that would have caught it.

4 caught 3 in code no test runs 3 run, but not checked 6 partly run
Bug list frozen
Before the run, 25 Sep 22:03 ET
Your suite
The project’s unit tests, under 2 seconds
Baseline
5 clean runs, 0 failing tests before any bug was planted
Every catch
Re-run once on the same code; 4 of 4 reproduced

In code no test runs

3

A coverage run on your unmodified code shows none of these lines execute under your tests. Any bug here ships silently.

#1

With no webhook secret configured, the webhook accepted an unsigned body and stored the event.

src/billing/client.ts:455–492 · the code today answers 500 in this case

Test to add: call the webhook route with the secret unset and an unsigned body; expect a 500 and no stored event.

#2

An error inside an event handler was logged and swallowed; the webhook answered 200, so Stripe would never retry.

src/billing/client.ts:518–520

Test to add: register a custom handler that throws; expect a 500 response.

#5

A new Stripe customer was created without the user-id idempotency key, so a retry could create a duplicate customer.

src/billing/client.ts:379–381

Test to add: mock the Stripe client and assert customers.create receives the user id as its idempotency key.

Run, but not checked

3

These lines execute during your tests, yet no assertion notices when they change. These are the cheapest gaps to close.

#4

Cancelling a subscription without the period-end flag cancelled it immediately, instead of the documented at-period-end default.

src/billing/client.ts:101

Test to add: cancel a subscription with the flag omitted; assert Stripe receives an update with cancel_at_period_end: true, not a cancel.

#10

A cancellation scheduled for the middle of the period was recorded as "cancels at period end".

src/billing/sync.ts:155

Test to add: sync a subscription whose cancel_at is a week before the period end; expect it not to be marked as cancelling at period end.

#15

Updating only a subscription's metadata wiped its org and user link, undoing a fix from an earlier release.

src/billing/api.ts:385–386

Test to add: update metadata without passing org or user; assert both keep their values.

Partly run

6

Some of the edited lines execute, but not the branch that changed.

#3

The customer-created and customer-updated handlers were swapped: new customers were never stored and updates were ignored.

src/billing/client.ts:545–546

Test to add: deliver a customer.created event; expect a stored customer.

#6

A subscription update stored the time it was cancelled in place of the time it will cancel.

src/billing/client.ts:594–596

Test to add: deliver customer.subscription.updated with cancel_at set; assert the stored value matches it.

#7

A new subscription's period end was stored in milliseconds while every other path uses seconds.

src/billing/client.ts:574–578

Test to add: create a subscription from an event, read it back, and compare the period end to the event's value.

#11

A deleted subscription was marked "cancelled", which no status check matches: Stripe spells it "canceled".

src/billing/sync.ts:305–306

Test to add: deliver customer.subscription.deleted; assert the status is exactly "canceled".

#13

A redelivered checkout-completed event inserted a second row, and later lookups that expect one row failed.

src/billing/sync.ts:336–337

Test to add: deliver the same checkout event twice; expect exactly one row.

#16

Listing a user's subscriptions searched the organisation index, so users saw none of their own.

src/billing/api.ts:153–154

Test to add: create a subscription for a user, then list by that user's id; expect it back.

Caught

4
  • #8Payment amounts stored in dollars instead of cents.
  • #9A caller's checkout parameters overriding the checkout mode.
  • #12A paid invoice pushed back to open by an out-of-order event.
  • #14A deleted customer's email kept on record.

Your score, month by month

This run is month 1. From next month, all 16 of these bugs are planted again, so each miss that gets a test turns into a catch, and up to 16 new bugs join them in the code you changed.

Month 1 · this run
4/16
25% caught16 new
Month 2
?/32
your 12 misses planted again+16 new
Month 3
?/48
+16 new

Share your score

When your score is one you want to show, we switch on a public receipt page and a badge for your README. They show the score, the date and how it was measured, never the misses. It stays off until you ask: reply to our email.

How this was measured

The 16 bugs were written from your code and docs without reading your tests, then frozen and hashed before anything ran, so the list could not be bent to the result. Each bug was planted alone in a private copy of your repository and your own test command was run against it. A bug counts as caught only when a test that passes on the unmodified code fails, and fails again on a second run. Your code is deleted within 30 days after you stop. Security and data.