Can your own history be your benchmark? Six steps from fixed issues to a test bench you can trust

Most choices about coding agents are made on public benchmark scores. A model or a harness tops a leaderboard built from other people’s repositories, and teams switch. What a technical leader needs to know is narrower: how does this setup do on your own code, with your own tests, on the kind of issue your team actually files?

The test set for that already exists in most teams. Every fixed issue that came with a test is a small exam with a known answer: the issue is the question, and the test that arrived with the fix is the marking scheme. The idea is not mine; it is how the SWE-bench benchmark was built. What I wanted to know is whether a team can do it small, on its own repository, in a weekend, and trust the result.

This post is the method as six steps a team runs on its own history. Then I run it on a public repository, to check that it holds up and to get a rough sense of what to expect. The repository is not the point and neither is the agent; the method is. The bench it produces is also the one the rest of this series runs on.

Step one: mine fixes that close one issue and change a test

Keep a fix only if it closes exactly one issue and changes a test file. The issue is all the agent will be given, so it has to be one issue, written by a person, with a real description. Read the closing reference from the pull request text and ignore anything inside comments: one repository I tried quotes an example “fix #123” in its pull request template, which made most of its pull requests look as if they closed the same issue.

One task per issue. A backport of a fix to a maintenance branch closes the same issue again. Keep the pull request into the main branch and drop the copies, or the same exam counts twice.

Start after the model’s training cutoff. Only fixes merged after the model you are testing stopped learning, so it cannot have seen the answer. On a private repository this matters less, and it costs nothing to keep.

Step two: split each fix

Withhold the test part. Every changed file under the test folders goes into a patch the agent never sees. The rest is the code part, kept only to check the task.

Step three: check the tests fail before and pass after, three times

Run it in a clean container, from the commit before the fix. Apply the test part and run the touched test files: something must fail. Add the code part and run them again: everything that failed must now pass. The tests that went from failing to passing are what a fix has to achieve; the tests in the same files that passed both times are what it must not break. A new test file that cannot even be imported before the fix counts as failing for every test in it.

Do it three times. A task whose result changes between repeats is flaky and goes. So does any candidate whose tests already passed before the fix, or did not all pass after it.

Step four: screen the tests against the issue

Ask of each task whether a fix written from the issue alone could pass its tests. Tests that came with a real fix often check more than the issue says: they import a helper the developer created and named while fixing it, or assert on the exact wording of a new message. That is fine for a regression test and useless for an exam, because no one who has only read the issue can match it. A task like that measures guessing.

Label each task, then check a sample by hand. A model can do the first pass from a packet holding the issue, the names of the must-pass tests and the withheld test diff, and nothing else, so the label cannot be swayed by how any run went. A person then reads a handful and agrees or overrules. I added this step after my runs, because of them; it belongs in the method from the start.

Step five: freeze the set

Write every task into one manifest and record a checksum. The commit, the issue text, the two lists of tests, the container image. Nothing changes a task afterwards, so two runs a month apart are comparing the same exam.

Step six: run your setup and score with the withheld tests

Give the agent the issue and the checkout, nothing else. When it stops, take everything it changed, put the withheld tests back as they were in the real fix, and run the two lists. Resolved means every must-pass test passes and no must-keep test breaks.

The first time, run it twice with nothing changed. The difference between two identical runs is the noise any comparison of setups has to beat, and it is worth knowing once, before anyone argues over a few points. If the two runs agree, stop repeating and spend on tasks instead.

What I wrote down before running it

My thesis was that a team can build this in a weekend, and that most candidate issues would not survive the checks. Five predictions, so that it could fail in public. Fewer than half the candidates would survive. The fail-then-pass check would remove the most, more than broken environments or flaky tests. Flakiness would be real but small, between one candidate and one in ten. Two identical runs would disagree on at least 15% of tasks. And within a run, the costliest resolved task would cost at least three times the median one. The fourth was the one I cared about most, because the next posts compare setups.

How I tested the method

A well-tested public repository. pytest, with every pull request merged from July to October 2026. My first choice was another large Python project, but it now builds a compiled core from source inside the repository, which a plain container cannot do cheaply. That is worth knowing about your own codebase before you start.

One plain setup. mini-swe-agent, a deliberately minimal agent from the team behind SWE-bench, with its own prompt unchanged, driving Claude Sonnet 5.5, whose training data ends in June 2026. Each task was capped at $1 and 75 model calls. This is not a comparison of tools; it is one setup, used to see whether the bench works and what a plain one scores.

What the run showed

The cull happens before the checks. Of 197 merged pull requests, 51 closed exactly one issue and changed a test, 48 once backports were folded in, and 44 passed the fail-then-pass check three times. Verification removed almost nothing and found nothing flaky, and each container took seconds. That is not a mark against the project; plenty of good pull requests fix something nobody filed. It is a fact about what a bench can use: on a repository whose maintainers insist on tests, the limit is how often a fix is linked to an issue at all, here about one in four. Expect a lower share where linking is a habit rather than a rule.

Half the survivors test something the issue never says. The screen labelled 23 of the 44 tasks not fair, most often because a new test imports a name that exists only in the real fix. On the fair tasks the agent resolved 62% in both runs; on the rest, about one in ten. The headline third is mostly a measure of the tests, not of the setup. The people who built SWE-bench Verified found the same problem at scale and screened every sample by hand: their annotators flagged more than half, and 500 of the 1,699 they labelled were kept. One well-tested project gave half; plan for that order.

Two identical runs agreed on 43 of 44 tasks. The paths differed (the same task cost up to 2.8 times more in one run than the other) but the outcomes did not. With 44 tasks the uncertainty sits in which tasks you happened to pick: a resolved rate of 34% on 44 tasks could plausibly be anything from about 22% to 49%. That is the gap two setups have to clear before the difference means anything, and a second run does nothing to narrow it.

Cheap, quick, and lumpy. Building the bench took no model calls and under an hour of compute. The two runs cost $16.58, with a median of 11 to 14 cents a task and the costliest resolved task 5.1 and 7.5 times the median. Each run took well under an hour. And 28 of the 29 misses in the first run were handed in as finished: the agent rarely knows when it has failed, so the tests have to.

Of the five predictions, two held and three failed. Far more than half survived; the fail-then-pass check did remove the most, three candidates against one, too few to mean much; nothing was flaky; the runs disagreed on 2% of tasks, not 15%; and the cost tail was as long as I expected. The first half of the thesis held with room to spare. The second half was wrong on this repository, and the step I had not written down, the screen, turned out to be the one that halves the set.

What I’d tell a fellow technical leader

Make the issue link and the test part of done. The bench can only be as large as that habit. Here, three in four merged fixes could not become tasks, and none of the checks after mining came close to that loss.

Screen your tests before you trust a score. Half of these tasks tested a detail no one could guess from the issue. Without the screen, a setup’s score says as much about how your team writes tests as about the setup.

Compare setups on the same tasks, and buy more tasks before more repeats. A second identical run changed one result in 44. The width of the uncertainty comes from the size of the task set, so a comparison of setups should pair them task by task and grow the bench, not rerun it.

Budget per task and watch the tail. The median task was cheap and a few were not. A per-task cap keeps one stuck attempt from deciding the bill, and the share of tasks that hit the cap is a number worth tracking.

Pick what your bench can build. A compiled core inside the repository, or tests that need services, multiplies the cost of every step. Start the bench where a plain container is enough.

Try it yourself

The kit now includes bench.py, with one subcommand per step: mine, image, split, verify, freeze, packets, run, score and report. It needs Docker or OrbStack, a read-only GitHub token for mining, a recipe file that says how to install your repository and which folders hold its tests, and the AI gateway key the kit already uses for the runs. At this size, mining and verification took under an hour and no model calls; each run took under an hour and cost under $10. The README has the recipe format, the fairness screen as a reusable prompt, and a short checklist for choosing a harness to plug in: who makes it, its licence, whether it runs unattended, whether it reports cost per task, whether spend can be capped, and whether it takes your own model endpoint. The bench measures what the tests check, not code quality or review effort, and one repository with one model is a check of the method, not a ranking.

What’s next

The series has now built every instrument a delivery line needs: lanes for triage, how much data a threshold needs, which tickets are worth an attempt, and a bench to try them on. The next post puts them on one line.

Appendix: method, data and limits

The study was written down on 31 October 2026 before any data was fetched, and every change after that was recorded with its reason before the result it affects was read. The main ones: the first repository was dropped before verification because of its compiled core; comments are ignored when reading closing references, after the template problem above; the one-task-per-issue rule was added after the verification counts were seen and before any agent ran (counted as a step instead of a mining rule, the second prediction would have tied rather than held); and the fairness screen was added after both runs. Its rule was fixed before any label existed: fair when the tests check behaviour the issue states or clearly implies, not fair when a must-pass test depends on a name, signature, message or choice the issue neither states nor implies. The labels were made by fresh Claude agents that saw only the issue, the test names and the withheld test diff, never the fix or any result; they come from the same family as the model under test, and I checked five tasks drawn at random by hand and agreed with all five.

Setup: mini-swe-agent 2.4.6 with its SWE-bench configuration, Claude Sonnet 5.5 through the Vercel AI Gateway’s OpenAI-compatible endpoint with the model’s default sampling, which made no use of prompt caching; cost counted from the reported token usage at list prices, which matched the gateway’s own figure. The harness and every dependency were installed only inside the task container, from a lock file of releases at least two weeks old. The agent ran in that container on a network whose only way out was a small proxy that allowed the model gateway alone, and its commands ran as a separate unprivileged user; the proxy logged 166 connections over both runs, all to the gateway. One lesson for anyone building the same: the harness writes its model settings into its run record, so a key passed that way ends up on disk; pass it some other way, and scan what a run leaves behind.

Limits: one repository, one model, short sessions (a median of seven model calls), 44 tasks. Resolved means the tests pass, nothing more. The model may have seen discussion of some of these issues before its cutoff even though the fixes came after it; for a team’s own private repository that does not apply.

Data: pytest is MIT-licensed; only public issues, pull requests and the repository’s own code and tests were used, and no contributor is named. I am not affiliated with the makers of pytest, the harness or the model, and none has reviewed or endorsed this post.

This is personal research, done in my own time and on public data only. The views are my own and not my employer’s.

I used Claude to help build the tool and draft this post.