Ask your own repository
How to evaluate your own AI setup against your own codebase, pipeline and history, one delivery step at a time: each post gives a method a team can run on its own repository, and uses public data only to check that the method works and to hint at what to expect.
Findings so far
What the method has shown across the posts, each tested on public history and meant to be checked on yours.
Measure on your own history. The same model, asked the same question, went from usable to untrustworthy between one repository and a public benchmark, so no published figure stands in for your own.
Check the room before the sample size. Whether a lane keeps its error promise depends mostly on how far under the budget the model's confident errors sit; more labels do not rescue a lane at the edge.
Grade against what a person confirmed. Labels set by bots or templates make a new model look like the old one, so record who set each label and measure against the ones people checked.
Correct for churn, and ignore the resolve button. A third or more of the review comments that look acted on would have changed anyway, and the resolve button runs 20 to 45 points ahead of the code.
Check the base rate before you build a sorter. When about a quarter of agent attempts land without a person stepping in, no signal read from a ticket can lift a slice to an unreviewed lane; raise the base rate first.
Screen the exam before you trust the score. Half the tasks built from one well-tested repository's fixed issues tested a detail the issue never stated, and two identical runs disagreed on one task in 44; screen the tests against the issue, then grow the task set rather than rerun it.
The posts
The posts in reading order. The first three were written before the series had a name and are its prologue; they build the three lanes and the habit of measuring against the team's own history that the later posts rely on.
-
How much of triage and code review can you hand to AI? A way to find out for your own team
Intake: the three lanes, prologue ·
Sort AI decisions into three lanes (no human, review later, human first) and choose each confidence threshold on one set of your own past decisions, then check it on another. On a public repository's 2026 history, triage could be partly handed over and code review could not, and the same model did far worse on public benchmarks than on that repository.
-
Can a cheap decision model take triage off your team's plate? What happened when Jev finally answered
Intake: confidence you can act on, prologue ·
The same lanes applied to a decision model that returns a probability rather than text, with six narrow review checks combined into one score. Its triage lane held on the repository's own history at about a twentieth of the cost, broke on noisy public labels, and no model earned a lane for code review.
-
Is it the model or your labels? What it takes to hand issue triage to AI
Intake: measure against labels a person confirmed, prologue ·
Grade AI triage only against labels a person confirmed, read from your issues' label history, to separate what a better model fixes from what was never written down. On one project the team's own labels did not help, routing learned from history beat every model, and a stronger model paid off only where the definitions existed.
-
How many tickets do you have to label before you can trust an AI lane? Fewer than you think, or no number is enough
Trust: measure the room before counting labels ·
Before deciding how many tickets to label, measure how much room the model has under your error budget among the calls it is confident about. With room to spare, 25 to 50 labels were enough; at the edge of the budget, the promise broke about half the time whatever the sample size.
-
Which tickets can you hand to an agent? Three numbers to pull from your own attempts before you try to sort them
Hand-off: three numbers and one division ·
Count three outcomes per agent attempt (merged, landed clean, still standing after 30 days) and divide your error budget by the landed-clean rate before you try to sort tickets. On 1,194 public tickets, 27% landed clean, which settled that no ticket-only sorter could meet a 15% budget.
-
Can your own history be your benchmark? Six steps from fixed issues to a test bench you can trust
The bench: six steps and a screen ·
Turn merged fixes that close one issue and change a test into tasks, check in a clean container that their tests fail before the fix and pass after it, screen the tests against the issue, freeze the set, and run your agent setup. On 197 pytest pull requests this gave 44 tasks in an afternoon with no model calls; half of them tested details the issue never stated, and two identical runs, $17 together, agreed on 43 of 44.
-
Do your review bot's warnings turn into bugs? What your history can and cannot tell you
Review: four numbers against a placebo ·
Judge an AI reviewer by four numbers from your own pull requests, with your human reviewers as the yardstick and unflagged lines from the same pull request as a placebo for churn. On three open-source projects people were heeded more per comment, the resolve button flattered the bot by 20 to 45 points, and whether ignored warnings became bugs needed a sharper bug signal than public history has.
The series continues. Every post comes with its code in ai-triage-review-eval, so the method can be run on your own repository. Personal research, my own views, not my employer's.