Same repository, bugs removed

I ran one bug-finding agent on two copies of the same repository. One copy still held the benchmark’s planted bugs. The other had every planted bug already fixed. The agent filed 68 reports on the first copy and 67 on the second.

SWE-sweep^[1] is a benchmark from Meta FAIR. The agent receives a repository and no issue text, and is asked to find and fix whatever is broken. The score is the share of 4,068 reference bugs, across 100 repositories, whose hidden tests the agent’s patch makes pass. The best model in the paper resolves 4.7%.

Recall against that list counts planted bugs the agent hit. A paired control can also ask whether the agent behaves differently once those bugs are gone, and whether an execution-free judge matches the code. In this note I measure both.

The comparison covers 46 paired runs and five models from two vendors. The agent found 9 of the 557 reference-bug opportunities it was given. Three findings carry the note.

• Report counts barely moved after the 61 reference bugs were removed. The agent filed 68 reports with the bugs present and 67 with them gone.
• The final estimate of bugs still undiscovered moved by −0.1 after those 61 bugs disappeared.
• An LLM judge of the kind the paper uses to count attempted fixes did not track what execution found.

Each SWE-sweep repository is pinned to a commit where 10 to more than 100 historical bugs are present at once. The agent gets a prompt that defines a repository-discoverable bug, a shell, and up to a thousand steps. Afterwards, each reference bug’s hidden tests are run against the patch. A bug counts as resolved if its tests pass and the repository’s own visible tests do not regress. The headline number is the fraction of all 4,068 bugs resolved.

The paper leaves work outside the reference set unpenalised. In its words, that work “may address code quality improvements, or address genuine bugs that are absent from our ground truth.” It then adds a looser count, which is meant to track a model’s broader tendency to find bugs. An LLM judge reads each final patch and lists every bug it attempts to fix. GPT-5.6 Sol attempts around four thousand by this count, for 191 resolved.

The paper notes that the judge “does not decide whether a fix is correct or complete” and that the count “is not checked by execution.” I wanted two measurements under that gap. Does an agent behave differently on a repository that actually contains the bugs? Is an execution-free judge a reasonable proxy for whether a reported bug is real?

Two copies of each repository

I picked five repositories whose reference fixes apply cleanly on top of each other: click and dateutil in Python, luxon in JavaScript, gin in Go, and syn in Rust. Together they carry 61 reference bugs. The buggy tree is the benchmark’s base commit with the .git directory removed, which is the tree the benchmark agent sees. The fixed tree is the same commit with the source-code half of every reference fix applied. I dropped the test half. No hidden test enters either tree.

On the two Python repositories I could check the pairing directly. The 23 hidden tests fail 23 times on the buggy tree and pass 23 times on the fixed one. The published patches’ line numbers do not refer to the base commit. More than half the hunks are offset, by a median of ten lines and up to 1,379. The patches still apply. I located every hunk by its content before grading anything by line.

The agent and the checkpoints

The agent can list directories, read files, and grep. It receives the benchmark’s own definition of a bug, copied from the task prompt, and the same instruction to find as many as it can. It files reports through a report_bug tool with a file, a line range, a description, and a confidence. Every ten to twenty exploration calls, and again when it decides to stop, it answers an estimate call. The call asks how many bugs it believes are still undiscovered, its probability that none remain, and its probability that every report it has filed is genuine.

I ran GPT-5.4, GPT-5.4 mini, GPT-4.1 mini, DeepSeek V4 Pro, and DeepSeek V4.1 Flash. Budgets were 50 to 80 exploration calls. That is 92 runs, of which 46 are buggy-fixed pairs that share a repository, model, budget, and seed. The runs cost about $16.

A report counts as finding a reference bug only if two checks pass. The reported lines overlap a located gold hunk, and an independent GPT-5.4 judge agrees that the report describes the same defect as the gold diff. Every report also received a standalone verdict from the judge, which read only the code around the reported lines. For the 73 reports on the Python repositories, I ran any test the report cited. GPT-5.4 wrote a small reproduction script, and I executed that script in the tree the agent had seen.

Report rate does not depend on whether the bugs are there

Removing all 61 reference bugs changed the aggregate report count by one. Fifteen pairs filed more reports on the buggy tree, nineteen filed more on the fixed tree, and twelve tied. GPT-5.4 was the only model that leaned toward the buggy tree, with 20 reports against 13. It was also the only model whose reports touched the reference set at a non-trivial rate.

Six distinct reference bugs were found in 46 buggy-tree runs. A gettz helper in dateutil strips name[:-1] where it should strip name[1:]. Three of the five models found it, which makes it the easiest bug in the sample. A missing quarters-to-seconds factor in luxon’s duration matrix was found by both DeepSeek models and by neither OpenAI model. Mean stated confidence across all 135 reports was 0.78. On the buggy trees, 9 of 68 reports matched a reference bug.

The agent believes the same thing on both trees

This is the result I trust most. At the final checkpoint, each model’s estimate of how many bugs remain is a model-specific constant. GPT-5.4 sits near one. DeepSeek V4 Pro sits near three. The constant stays put when the bugs are removed. On the buggy trees the models under-estimate the remaining bugs by five to ten times. On the fixed trees they still report one to three.

Every one of the 92 runs stopped voluntarily, after a mean of 44 exploration calls out of a budget of 50 to 80. The paper reports that its agents also stop early. They spend a median quarter to a third of their steps after the last bug they resolve. The checkpoints add a fact to that pattern. The decision to stop is made with the same stated beliefs on both trees. Stated probability that all filed reports were genuine ranged from 0.70 to 0.97. Reference-set precision on the buggy trees was 9 in 68.

The judge agrees with the agent, and the code does not

For the 73 reports on click and dateutil, I had two independent readings of each claim. The GPT-5.4 judge, reading the code, labelled 33 of them genuine bugs. A reproduction script, actually run, found that the described behaviour existed for 15 of those 33, did not exist for 7, and could not be checked for 11. Under the judge’s label of not a bug, the behaviour reproduced more often than under the label genuine bug.

A behaviour that reproduces can still be the design. The clearest case was a GPT-5.4 mini report, filed with 0.96 confidence, that click ignores environment variables set to the empty string. The judge called it genuine. The script confirmed the behaviour. The one test the agent itself had cited, test_empty_envvar, passes. It asserts exactly that behaviour. An empty MYPATH yields None. The repository’s own test suite treats this as the design.

This is the same procedure the paper uses to count four thousand attempted fixes. In this sample the judge’s label barely predicted whether the described behaviour existed. It predicted even less about whether the behaviour was wrong.

A control is useful if the fixed tree is clean. This one is not. Reading all 67 fixed-tree reports, a few findings recur across runs and model families. The recurrence is the useful part.

A real bug worth zero points

Luxon’s DateTime.fromHTTP calls parseDataToDateTime(vals, parsedZone, opts, "HTTP", opts). Its sibling methods pass the input text as the last argument. This method passes the options object. The error message for an unparseable date shows [object Object] in place of the input. Three model families reported it, eight times in total, on both trees. The bug was introduced in luxon 1.20.0, the release the benchmark pins, and was fixed upstream in April 2026,^[2] after the benchmark was built. It is a real, maintainer-confirmed defect. The reference score gives it no credit.

dateutil has a similar defect. In _tzicalvtz._find_comp, a fallback branch does lastcomp = comp[0]. That indexes the loop variable, a component object, where it should index self._comps[0]. It raises a TypeError whenever every component is daylight time and the queried datetime precedes the first transition. Three runs found it. It is still on master today.^[3]

• luxon · DateTime.fromHTTP passes opts where text belongs8 reports · 3 model families

  Real. Fixed upstream in moment/luxon #1775, April 2026. Not in the reference set.

• dateutil · _find_comp indexes the loop variable3 reports · 3 models

  Real. Still present on master. Not in the reference set.

• dateutil · tz*.__ne__ breaks the ComparesEqual tests5 reports · GPT-5.4

  False. Contradicted by execution; the cited tests are in the benchmark’s own passing list.

• click · empty environment variables are ignored3 reports · GPT-5.4 mini

  True and intended. test_empty_envvar asserts the behaviour.

• syn · generic arguments are printed lifetimes-first3 reports · 2 models

  True and intended. The comment above the loop says so. Judge: genuine bug, three times.

• gin · an “extra” block in findCaseInsensitivePathRec1 report · DeepSeek Flash

  The block is the reference fix. The agent flagged the fix as the bug; the judge agreed.

A confident wrong report

On the fixed dateutil tree, GPT-5.4 filed five reports saying that tzutc.__ne__, tzoffset.__ne__, and their siblings break the repository’s ComparesEqual tests. Each method is written as return not (self == other). The judge called four of them genuine. The claim is false. When __eq__ returns NotImplemented, self == other falls back to the reflected ComparesEqual.__eq__, which returns true. Then __ne__ returns false, which is what the test asserts. All three testEquality tests are in the benchmark’s own list of tests that pass at the base commit. The agent quoted the test as its evidence and could not run it. The judge, reading the same code, endorsed the story.

Removing the reference bugs did not change what the agents reported. The reports were never drawn from the reference set. On both trees, 123 of 135 reports fell outside it. That set mixes real bugs the benchmark cannot credit, design decisions it cannot penalise, and false claims its judge accepts.

Before settling on the control, I audited the released task data in a few other ways. None of these changes the paper’s qualitative story. Each one matters if you use the benchmark.

Seventeen of the hundred repositories ship with an empty list of visible tests that must keep passing. For those repositories, 1,224 bugs, or 30% of the benchmark, the regression check cannot fail. Any patch clears it, including a destructive one. A deliberately destructive patch I wrote for syn cleared the check and still scored one bug out of thirteen.

One hundred and fourteen groups of reference bugs share a byte-for-byte identical source fix once line offsets are ignored, 236 bugs in total. One hundred and eleven of the groups also share identical hidden tests. Fixing one fixes the group. The paper’s duplicate filter compares raw patch bytes and catches one group. The excess is 122 bugs, about 3% of the benchmark, concentrated in numpy, astropy, and pandas.

With a repository-cluster bootstrap, five of the nine adjacent pairs on the leaderboard are not statistically separable. Between 55% and 75% of every model’s score comes from repositories with fewer than 25 bugs. The published oracle patches also carry line numbers from the upstream pull request, not from the base commit. Any analysis that uses those numbers as hints or distances inherits the offset.

The strongest counterargument is scale. My agents had 50 to 80 read-only tool calls. The paper’s agents had a shell and a thousand steps, and could have run the tests that would have rejected the __ne__ reports. I agree. The execution result is an argument for that larger budget. The calibration result is about stated beliefs. GPT-5.4 filed more reports when the bugs were present, 20 against 13, and still did not move its estimate of bugs remaining by one bug when 61 bugs were removed.

A recall score against a reference list misses that split. The paper’s attempted-fix count and its discussion of early stopping both rely on a judge that, in this sample, did not track execution. Three checks would measure the rest. Run the test a report cites, and record whether the edit makes that test pass or breaks a visible one. Ask the agent, at submit time, how many bugs it thinks remain and how sure it is that the repository is clean, and report those numbers beside recall. Run a no-known-bugs arm, as the fixed tree does here, and treat its report rate as a false-alarm baseline. Finds on that arm that the reference set misses, such as luxon’s fromHTTP, are also candidates for new reference bugs.

The sample is 92 runs, five repositories, and read-only tools. None of the five model configurations tops the paper’s table. One LLM judge made errors in both directions. Execution checks cover two of the five repositories. The fixed tree means known bugs removed, not that the repository is correct. The paired design, the sign tests, and the execution check keep the result from resting on any one of those limits.