How much of triage and code review can you hand to AI? A way to find out for your own team

Over the past year, the speed at which teams produce change has gone up noticeably. Coding agents draft features, fix bugs and open pull requests, some of them while nobody is watching. Writing code is no longer the slow part.

The slow part is what comes after. Someone has to decide what an incoming issue is and who should pick it up. Someone has to review a change before it ships. Both need a person’s judgement, and most teams I talk to have little trust in handing either to AI. QA sits in the same place. So the work that grows fastest lands on the steps teams are least willing to automate.

I don’t think the answer is to hand everything to a model, or to keep every decision with a person. It is somewhere in between, and I wanted a way to find out where.

A new kind of tool

Most of us reach for a large language model when we want AI to make a call: give it the issue, ask what it is, read the answer. A newer kind of tool works differently. Decision models such as TypeSafe’s Jev don’t write an answer; they answer a narrow, typed question (a choice, a score, a yes or no) with a probability attached, at a fraction of the cost of a language model.

The price is attractive, but what caught my attention is the probability. If a model says it is 95% sure and is right 95% of the time, you can let it make the confident calls on its own and route the rest to people. That is exactly what triage and review need. Early independent work also suggests the result depends heavily on how the question is asked: one published test went from 63% to 95% accuracy by splitting one broad question into five narrow ones and combining the answers with a small model fitted on labelled examples.

I planned to test Jev first. I couldn’t, at least not yet. Over a full day of retries, seven requests got through. Everything else came back with “the upstream provider is currently experiencing high demand”, from both hosts that run it. When it did answer, it took about a quarter of a second and cost a fraction of a cent. That is not a criticism of a young product that a lot of people want to try at once, but it is worth knowing if you plan to put one in your delivery pipeline: availability is part of the evaluation.

So Jev waits for a second post. What I could do meanwhile was build the way I would evaluate it, and try it on the options available today.

How I evaluate it

A better question. “Is AI good enough to review code?” has no useful answer. A better one is: what share of these decisions can go without a person, at an error rate a team accepts? I think of it as three lanes. The first is no human: the model decides alone, and for this study I allowed about one wrong call in twenty (5%). The second is review later: the model decides and a person checks afterwards; I allowed 15% here. The third is human first, for everything else.

A model earns a lane only if its confidence separates the calls it gets right from the ones it gets wrong. So for each model I chose a confidence threshold on one set of examples and then checked it on a separate set it had never seen.

Your repository already has the answers. Every labelled issue records what your maintainers decided it was. Every merged pull request records whether a reviewer commented or asked for changes. That is free, labelled data about your own team’s judgement, and it is the fairest test there is. For this post I used HashiCorp’s Terraform, a large, well-run public repository, and only its 2026 history, so the models I tested are unlikely to have seen it in training: 203 issues its maintainers labelled as bug, feature or question, and 400 merged pull requests, 57 of which drew review comments or change requests.

Public labelled data as a cross-check. I also ran the same questions on two public research datasets: 1,500 issues from five open-source projects (the NLBSE 2024 issue classification set) and 1,000 code changes, half of which drew a review comment (Microsoft’s CodeReviewer data). They are older and not specific to any one team, but they show whether a result holds beyond one repository.

What I tested. Three options you could run today. Claude Haiku 4.5, a fast and inexpensive language model. Claude Opus 4.7, a larger one at roughly six times the cost per decision, on a 1,000-item sample of the public data. And an open zero-shot classifier (DeBERTa-v3, from Microsoft, adapted by Moritz Laurer) running locally on my laptop, free and private. Each got the same question wording, fixed before I saw any results. The whole study cost about $7 in API fees.

What I found

On Terraform’s own history:

For comparison, a rule that says “no reviewer will comment” on every pull request would be right 86% of the time.

Triage looks promising. Haiku sorted 92% of Terraform’s issues correctly, and its confidence was good enough to leave about four in five of them to it, at an error rate just over my 5% aim. The local classifier managed half, at a little more error. For a team that triages every issue by hand, that is real time back.

Review still needs people. Terraform’s reviewers commented on about one pull request in seven. Haiku flagged more than one in three, and still missed about half of the ones reviewers did comment on. The local classifier flagged nearly all of them. Neither was confident in the right places, so neither earned a lane. A tool like this would add review work rather than save it.

The same model looks very different on different code. On the public issues, the same Haiku with the same question sorted 72% correctly and could not be trusted with any of them alone. On public code changes it was close to a coin toss. Terraform’s issue templates make bugs and feature requests look different from the first line, and that matters more than the model. It is the strongest argument I found for measuring on your own history rather than trusting someone else’s benchmark, mine included.

Confidence matters more than accuracy. On the public issues, Opus 4.7 was only a few points more accurate than Haiku (77% against 74% on the same sample), yet its confidence was good enough to leave about 39% of issues to it, where Haiku got none. You pay roughly six times more per decision for that. The local classifier shows the opposite: when it said it was 99% sure, it was wrong about a quarter of the time.

Thresholds need a margin. Every no-human lane in this study was set for 5% error on one set of examples and landed at 6 to 8% on new ones. The sets I tuned on were small (61 issues for Terraform), and small samples flatter.

What I’d do as a CTO

Start with triage, not review. Triage is where the models were closest to useful, and a wrong label is cheap to fix. Review is where they would currently add work.

Judge a model by its confidence, not only its accuracy. Two models with similar accuracy can be worlds apart in what you can safely hand them. Ask vendors how their confidence is calibrated, and check it on your own data.

Set thresholds with a margin. If your team can live with 5% error, aim for 3% when you pick the threshold, and pick it on a few hundred labelled examples, not a few dozen.

Keep a review-later lane. Much of the value sits between “no human” and “human first”: the model decides, and a person spot-checks afterwards. It builds trust with real evidence instead of asking the team for a leap of faith.

Re-measure regularly. Models change, your repository changes, and so do your team’s habits. If the measurement is cheap, repeat it every quarter.

Try it on your own repository

Everything above is in a small kit: ai-triage-review-eval, under the Apache 2.0 licence. It fetches your 2026 issues and merged pull requests through GitHub’s API with a read-only token, asks each model the same questions, and reports the three lanes. It needs only standard Python, an API key for the models, and optionally Docker for the local classifier. On Terraform, Haiku took about 20 minutes and cost $1.30 for 603 decisions. The local classifier runs for about an hour on a laptop.

The results are only as good as your team’s labelling and review habits. An approved pull request may still have drawn a comment somewhere, and very large changes are left out. The kit’s README covers these limits.

What’s next

When Jev is reachable at scale, I’ll run it through the same kit, on the same data, with the same questions, and look at whether narrow, well-designed questions close the gap on code review. That is the second post.

Appendix: method, data and limits

Everything was fixed before any results were seen: data selection, question wording, metrics and the 5% budget. The review-later lane was added later, before results, and is exploratory. Scores use a held-out evaluation set; thresholds are chosen on a separate calibration set. Failed or unreadable answers count as wrong. I used Opus 4.7 because newer Opus versions were not available to my account; it ran on a 1,000-item stratified sample of the public data only. On Terraform, a pull request counts as “needed a comment” if any reviewer other than the author left a comment review or requested changes (Terraform’s reviewers almost never use “request changes”); pull requests with diffs over 20,000 characters were excluded. The local classifier reads at most 512 tokens, so it saw only the start of most pull requests.

Data: the NLBSE 2024 issue report classification set (issues from React, TensorFlow, VS Code, Bitcoin and OpenCV); Microsoft’s CodeReviewer data (Li et al. 2022, CC BY 4.0); hashicorp/terraform issues and pull requests created in 2026, fetched through the GitHub API, reported in aggregate only. Models: Claude Haiku 4.5 and Claude Opus 4.7 through Vercel AI Gateway; DeBERTa-v3-large zero-shot v2.0 by Moritz Laurer (MIT licence).

This is personal research, done in my own time and on public data only. The views are my own and not my employer’s.

I used Claude to help build the test harness and draft this post.