How many tickets do you have to label before you can trust an AI lane? Fewer than you think, or no number is enough

Most teams I talk to now let a model take some decisions on its own: label this issue, route that one, wave a small change through. The sensible ones do not switch it on blind. Someone labels a sample by hand, checks how often the model is wrong when it is confident, and picks a cut-off that keeps errors under an agreed budget, say 5%.

That check is where the trouble starts. In my first post, every lane I set for 5% error landed at 6 to 8% on fresh data, mine included. The obvious fix is to label more tickets before trusting the number.

So I set out to measure how many is enough. The answer turned out to be a different question: how much room the model has under the budget decides far more than how many tickets you label.

The thesis I wrote down

A cut-off picked on a small sample promises more than it delivers, because choosing the most generous cut-off that passes favours lucky samples. Before running anything I wrote down four predictions: with 100 labels or fewer the promise breaks in more than half of the cases; more labels shrink the miss, but at 400 labels it still breaks in more than a third; a stricter rule with a statistical safety margin keeps the promise nine times in ten at a modest cost; and a cut-off set on last quarter’s tickets does worse on this quarter’s than random samples suggest.

Three of the four were wrong. What was left is more useful than what I predicted.

How I tested it

No new model calls for the first run. My earlier posts left me with about 28,000 model answers, each with a confidence score, on a public benchmark, Terraform and Metabase. For each model and decision I pooled every answer, drew a sample of 25, 50, 100, 200 or 400 labelled tickets, picked the cut-off the same way as in my earlier posts, and checked it on all the tickets the sample had not seen. I repeated that 2,000 times for each sample size. Here the models are only a source of confidence scores; what is under test is the procedure a team uses to switch a lane on.

Three rules. The plain rule is the one teams use: the most generous cut-off at which the sample’s error stays within budget. “Aim lower” targets 3% to get 5%. The safety margin accepts a cut-off only when a statistical upper bound on its error is within budget, which in practice means at least 45 confident answers with no mistake among them before a lane opens.

Time, not just chance. Random samples are the best case, because the sample and the future come from the same months. Where tickets have dates I also set the cut-off on the tickets just before a given day and tested it on everything after.

Then a test on fresh data. The pattern I report below was found after the fact, so I wrote down how to test it, then ran it on two projects the earlier posts had not touched. On PostHog that meant bug against feature on 1,383 issues, and routing issues to PostHog’s own team and product-area labels. On Sentry it meant routing 557 issues to its product-area labels. That took about 6,000 decisions from Claude Haiku 4.5 and TypeSafe’s Jev, and $2.91.

What I found

Of the first run’s 30 pools, 18 could support a lane at all. Sorting them by how much room the lane has under its budget explained nearly everything, and the fresh data agreed:

For comparison, simply sending every ticket through the lane is safe only when the model is wrong less often than the budget, which is the room column and nothing else.

With room to spare, 25 to 50 labels are enough. Where the model was wrong on 2% to 3% of tickets against a 5% budget, the promise held in every one of the 2,000 samples from 25 labels up, on Metabase and on fresh PostHog data alike. Much the same held at the 15% budget: the closest call, a decision with 14% errors, broke its promise in at most 6% of samples. Labelling more bought nothing.

At the edge, no number of labels is enough. Where the model’s error over all the confident tickets sits right at the budget, the plain rule settled at breaking its promise about half the time, in 42% to 53% of samples at the largest sample I could test, however many labels were used. In one case the rate went up, from 30% with 25 labels to 46% with 400, because a bigger sample finds the edge more precisely and then lands on either side of it. The typical miss is small, a point or so, but the worst tenth of samples landed between 5.5% and 10% against a 5% budget. On Sentry’s product areas, the two fresh edge cases broke their 15% promise in 47% and 49% of samples at 200 labels.

On a hard decision, small samples invent lanes. Where a model was wrong a third of the time or more, a lane rarely appeared in 50 labels, and when it did it broke its promise almost every time. Routing PostHog issues to teams or product areas, lanes found in 50 labels broke a 15% promise in 97% to 100% of cases, with typical errors of 21% to 27%.

Both fixes work at the edge, at a price. Aiming lower brought breaks down to 2% to 6% of samples in five of the seven edge cases, keeping 54% to 92% of the lane. The safety margin did about as well where the model reports fine-grained confidence. Neither works well with too few labels (on Terraform, with only 100, aiming lower still broke in a third of samples) or where a model reports a handful of round numbers such as 0.85 and 0.95, which makes a lane all or nothing: one model gave between 4 and 15 distinct values, and for it aiming lower shrank one lane to 8% of its size.

Lanes did not go stale with time alone. A cut-off set on earlier tickets broke its promise on later ones in 4 of 138 cases across Metabase, Terraform and PostHog, less often than random samples. That only covers the stable decisions, though. In my third post, routing to teams drifted because team ownership moved during the year.

What held and what did not. Fewer than half of samples broke the promise at 100 labels in three of five pools, so the first prediction failed. The second held only at the edge. The safety margin cut breaks to 6% to 14% where it worked, not to one in ten everywhere, and in some cases it barely opened a lane at all. Time made things no worse. On fresh data, the room and sliver patterns were confirmed by the rule I set in advance. The edge pattern held in both fresh cases I found, one short of the three I had required to call it confirmed.

What I’d tell a fellow technical leader

Measure the room before counting labels. Label 50 tickets and look at how often the model is wrong among the ones it is confident about. If that is well under your budget, switch the lane on; more labelling buys almost nothing.

At the edge, decide what your number means. If the confident error sits at the budget, it will break your promise about half the time whatever the sample size. Either treat 5% as a target rather than a guarantee, or aim for 3% and give up some coverage.

Distrust a lane found in a small sample of a hard decision. If the model is wrong a third of the time overall and 50 labels show a clean lane, it is most likely luck. Draw a second sample before believing it.

Check how your tool reports confidence. A handful of round values gives you a handful of possible cut-offs, so the lane is all or nothing and no clever rule rescues it. Fine-grained scores are worth more than they look.

Re-check after a change, not on a calendar. Nothing I tested went stale with time alone. What moved results in my earlier work was a reorganisation and a change of labels, so tie the re-check to those events.

Try it yourself

The kit now includes error_budget.py. Point it at the answers from your own run and it redraws your samples, tries the three rules and tells you whether each decision has room, sits at the edge or is a sliver. It needs only standard Python and makes no model calls, so it costs nothing beyond the run you already have. A busy repository can now be fetched with --via-search, for projects whose issue lists GitHub will not page through. The limits are the same as before: the answer is only as good as the labels you check it against.

What’s next

The next post in this series looks at review bots: how often their comments lead to a change in the code they point at, next to the same figure for human reviewers in the same repository, on public history and without a single model call.

Appendix: method, data and limits

The study was written down on 10 October 2026 before any analysis ran, and revised once before running, when the first draft read as a test of models rather than of the procedure. The grouping into room, edge and sliver was found after the first run; its definitions and three checks were then written down before any fresh data was fetched, along with the rule that a check counts as confirmed when it holds in at least four of five fresh cases, with at least three cases. Every later change, including the move from bug-against-feature to routing when the first fresh run produced only room cases, was recorded before the affected run.

Groups come from the plain rule applied to a whole pool: room when the lane keeps at least 99% of tickets, edge from 40%, sliver from 10%. A sample size counts only when at least 300 tickets remain to test on. The safety margin uses a one-sided 90% exact binomial bound; because it scans many cut-offs, its textbook guarantee does not strictly apply. Pools that share tickets are not independent, so I report ranges and no combined significance test.

Limits: the 5% lane had a thin base, five of 30 pools in the first run; random samples are a best case; the time check covers only decisions stable enough to have a lane; on PostHog most team and area labels were chosen by the reporter when filing, and on Sentry about two in five came from its triage bot, so both are noisier ground truth than a triager’s labels; PostHog labels almost no questions, so its issue decision is bug against feature.

Data: the NLBSE 2024 and CodeReviewer benchmarks, and 2026 issues and pull requests from four public repositories, fetched through the GitHub API and reported in aggregate only, with no issue authors or reviewers named. The fresh runs used Haiku and Jev through Vercel’s AI Gateway.

I am not affiliated with PostHog, Sentry, Metabase or HashiCorp, and none has reviewed or endorsed this post.

This is personal research, done in my own time and on public data only. The views are my own and not my employer’s.

I used Claude to help build the test harness and draft this post.