Which tickets can you hand to an agent? Three numbers to pull from your own attempts before you try to sort them

Coding agents now take tickets. You assign an issue, the agent opens a pull request, and someone decides whether it goes in. Most teams I talk to have settled into a habit of giving the agent the small, well-described things and keeping the rest, and the habit is set by feel.

That habit is a triage decision, made from the ticket before any work starts, and it is the one gate in delivery I had not yet measured. My earlier posts sorted triage and review decisions into three lanes: no human, review later, human first. The natural question is whether there is a fourth, an agent lane: tickets you can hand over and expect to land, within an error budget you set in advance.

This post is about how to answer that from your own history, in three steps that get more expensive as they go, so that you can stop at the first one that settles it. I tried the method on public tickets that agents had attempted, to check that it holds up and to get a rough gauge of what a team should expect to see. The tickets are not the point and neither are the agents; the method is.

Step one: three numbers per attempt

Keep the link from attempt to ticket. None of this works without knowing which issue each agent pull request was for. Most setups only keep that link when the agent happens to write “fixes #123” in the description. Make it a rule, because it is the one piece of record-keeping the rest depends on.

Count landed three ways. The loosest outcome is merged. Stricter is landed clean: merged, no reviewer asked for changes, and no commit from a person other than the agent, not counting merges of the main branch or other bots. Strictest is landed and held: landed clean, and no fixing commit on the same files within 30 days. The first number is what dashboards show. The gap between it and the other two is human work the agent did not save, and it is the number a renewal conversation should be about.

Decide what counts as the agent. An agent that opens pull requests from its own account makes this a comparison of logins. One that commits under a developer’s account does not, and unless its commits carry a reliable marker you can only judge it at merged. Write the rule down before you look at a single outcome.

Step two: one division

Divide your budget by your base rate. An unreviewed lane at a 15% error budget, the budget my review-later lane uses because a failed hand-off costs a person a look, needs a slice of tickets that lands 85% of the time. If a quarter of your attempts land clean, that is a lift of more than three times over handing everything over. No signal I know of that can be read from a ticket (task type, acceptance criteria, scope, a model’s judgement) produces separation like that; in my runs the best one roughly doubled the odds on its top tenth. So the base rate decides whether a lane is possible at all, and the sorter only decides how big. If the lift you need is three times or more, stop here: the money belongs in raising the base rate, with clearer tickets, better tests and tighter scope, not in sorting. Only once attempts land clean more often than not is step three worth running.

Step three: a sorter judged on later months

Read only the ticket. A model reads the issue title and body, the repository’s language and a rough size, nothing else: not the repository’s name, not the agent, not the pull request. It returns a task type, six yes or no checks (are there acceptance criteria, a reproduction, a bounded scope, an open design decision, an external dependency, a change across many parts), and its own probability that an agent would land it. From these come four rankings: a trivial rule (documentation yes, everything else no), task type alone, the model’s probability, and task type plus checks combined.

Make the lane earn its budget on tickets it has not seen. Split your attempts by date. On the earlier ones, the lane is the largest share of tickets, taken from the top of a ranking down, that stays within the budget under the safety-margin rule from my error budget post, which refuses to open on a lucky run of a few successes. Then judge that lane on the later tickets. A lane that holds on the months it was fitted to and breaks on the next ones is the common failure, and the split is what shows it.

Check it before you read it. Shuffle the outcomes within each repository and confirm no lane opens by chance. Run twice and confirm the output is identical. Pick ten attempts at random and check the tool’s reading of each against the pull request page. All of this is in the kit and takes minutes.

What I wrote down before running it

Five predictions, so that the method could fail in public. Documentation tickets land at least ten points more often than feature tickets. The trivial rule breaks the 15% budget. Task type plus checks opens a 15% lane covering at least a quarter of tickets. Task type alone gets two thirds of that coverage on its own. And nothing opens a 5% lane. The third was my thesis, and I expected the fourth to be the finding.

How I tested the method

Public tickets that agents attempted. AIDev, from a research group at Queen’s University in Canada, collects pull requests opened by OpenAI Codex, GitHub Copilot, Devin, Cursor and Claude Code in public repositories, and links some of them to the issue the agent was given. I kept pull requests with exactly one such issue, filed by a person before the pull request and with a real description, and created at least 60 days before the data was collected so each had time to resolve: 1,194 tickets from late 2024 to early June 2025, split on 22 May into 716 earlier and 478 later ones.

Most of it is one agent. Agents that work from assigned issues are naturally the ones that leave a ticket behind, and one of them accounts for about two thirds of this sample and nearly all of the later tickets. Two of the five commit under a developer’s own account, so for them I could only tell merged from not merged. Results are pooled across agents; this is not a comparison of tools.

Cheap, and checked. Scoring 1,081 distinct issues with Claude Haiku 4.5 cost $1.41. The shuffle check opened no lane, two runs were identical, and all ten pull requests I checked by hand matched the tool.

What the public tickets showed

The baseline is the last column: hand the agent everything, and about one ticket in four lands clean.

Step two settled it before step three ran. With 27% landing clean, a 15% lane needed a lift of more than three times, and the aggregate counts showed that before any ticket was scored. I ran step three anyway, to see how close a sorter could get. No ranking opened a lane at 15% or at 5%. The best of them, the model’s own probability, landed 55% of its top tenth and 64% of its top twentieth, against 85% needed. The third prediction failed, the fifth held trivially, and the fourth could not be judged without a lane to compare against.

Task type moves the odds, a lot, and is still not a lane. Documentation tickets landed clean more than three times as often as feature tickets, 57% against 17%, with intervals well clear of each other; the first prediction held, on a documentation cell of 37 tickets. But “documentation yes” hands over 8% of the later tickets and 43% of those do not land, nearly three times the budget. The second prediction held, uncomfortably: even the best category fails close to half the time.

The order of task types did not carry over. On the earlier tickets chores landed most often and documentation second; on the later ones documentation led and chores fell to a third. The later period is also a different mix of agents and the cells are small, so this is a warning about re-checking a prior when the mix changes, not a finding about chores. For the same reason I read nothing into the fitted combination of type and checks doing worse than the model’s single probability.

More than a quarter of clean landings came back. Of the later tickets that landed clean, 27% had a fixing commit on the same files within 30 days. Merged said 48%, landed clean 27%, still standing a month later 19%: each rung of the ladder is a different conversation.

What I’d tell a fellow technical leader

Count three numbers, not one. Merged, landed without a person stepping in, and still standing a month later. None of them needs a model, a budget or more than an evening with your pull request history, once the link from attempt to ticket is kept on purpose.

Do the division before you build or buy a sorter. Budget over landed-clean rate gives the lift you need. Three times or more, and no classifier, ticket template or prompt gets you there; spend on the base rate instead. This is the “measure the room” rule from my error budget post, and it would have saved me step three.

Until then, budget a human look for every attempt. The decision you can make from the ticket is which ones are worth an attempt, so price an attempt in reviewer time and compare it with doing the work yourself.

Treat task type as a prior, and re-check it when the mix changes. Documentation is a far better bet than features, and that held up. The ranking of the categories in between did not survive a change of period and agent mix, so recompute it after you change tools or take on a different kind of work.

Try it yourself

The kit now includes agent_lane.py. Name the accounts your agents open pull requests or commit from, give it a read-only GitHub token, and it builds the table of three numbers from your merged and unmerged agent pull requests and the issues they close; that step needs no model and is worth running on its own. If the division says sorting can pay, the same script scores each ticket through the AI gateway the kit already uses, at about $1.50 per thousand tickets with Claude Haiku 4.5, chooses lanes on your earlier months, judges them on your later ones, runs the shuffle and determinism checks, and prints the report above for your repository. It needs only standard Python. Expect the lane step to mean something once you have a few hundred attempts with an outcome.

What’s next

The next post builds the test bench the rest of the series runs on: turning a repository’s own fixed issues into a small private benchmark, and counting how few survive the checks that make a task trustworthy.

Appendix: method, data and limits

The study was written down on 24 October 2026 before any data was downloaded, and every change after that was recorded with its reason before the result it affects was read. The main ones: the issue-linked part of AIDev is much smaller than its headline figure (4,923 links, all from pull request bodies, in the dataset’s v3 release only), and after the 60-day rule it fell under the size I had set for enlarging it, so I published on the smaller sample rather than add data from a later release. Which commits count as the agent’s was decided per agent from account names and 50 sampled commits before any outcome was computed: three agents commit from their own accounts and are judged at landed clean; two commit under the developer’s account with no reliable marker and are judged at merged only, which is why the table’s ticket counts are slightly larger than the counts behind the landed rows. A fixing commit is one whose message starts with “fix” or “revert”, or that refers to the pull request’s number, excluding the pull request’s own commits and its merge; two pull requests no longer exist and could not be checked.

Scoring: Claude Haiku 4.5 at temperature 0, one call per issue, the issue truncated at 6,000 characters; eight answers named a task type outside the list and were left out. Task type read from the issue agreed with the type AIDev derived from the pull request title for 80% of tickets. The type-plus-checks ranking is a small logistic regression fitted on the earlier tickets with out-of-fold scores. Intervals come from 2,000 bootstrap draws over repositories. For comparison, Pinna and colleagues report 82% of documentation and 66% of feature pull requests merged across all agent pull requests in the same dataset; starting from a ticket and counting merges alone gives 75% and 40% here, and the ordering is the same. Leaving out each agent in turn changes no verdict, except that without the largest one only seven documentation tickets remain on the later side.

Limits: the model was trained on data up to July 2025, after every ticket here, so it may have seen some of these issues, and I could not run the planned check on unseen tickets because there were none; this does not apply to a team’s own private tickets. Landed clean counts any human commit as rework, including a one-line fix. Open-source maintainers decide differently from teams with a backlog and a deadline.

Data: AIDev by Hao Li, Haoxiang Zhang and Ahmed E. Hassan, licensed CC BY 4.0, with each source repository keeping its own licence. Only issue text, pull request metadata, commit authors and messages and review states were used; results are reported in aggregate, and no repository, author or reviewer is named. Agents are named only as the dataset names them, with no figure per agent.

I am not affiliated with the dataset’s authors or with any of the agents’ makers, and none has reviewed or endorsed this post.

This is personal research, done in my own time and on public data only. The views are my own and not my employer’s.

I used Claude to help build the tool and draft this post.