We wrote down what we were going to measure, then measured it.

What we measured

120 issues were selected from 8 public repositories across six languages, and 108 of them scored. Watari read each issue title and body, nothing else, and searched a single workspace holding all 8 repositories at once, 130,826 indexed chunks in total. Ground truth is the file the maintainer changed in the merged fix.

Why you can believe this one

Every case above is out-of-sample. All 8 repositories are new to this benchmark, and not one of them was examined while choosing any change we have shipped. That was not true of the previous run, and we said so at the time.

The run before last measured 57.1% (24 of 42) on the cases it had never examined, across six different repositories. This run measures 56.5% (61 of 108, 95% interval 47.1% to 65.5%) on 108 cases from eight repositories that share no case with it. Two measurements of the same claim on different data, agreeing to within a point. If they had disagreed, both would still be on this page.

The last run reported 64.3% (36 of 56) and we published it as a ceiling, because the indexing change it measured had been chosen by looking at those same cases. Out-of-sample it lands lower, which is what a ceiling means and why that figure was never the one to quote. It is not withdrawn.

The interval is the other thing that moved. Sample size is the only lever on it, and going from 56 scored cases to 108 narrows it from roughly 25 points to 18.4. It is still wide, and we would rather show you a wide interval than a point estimate without one.

What we knew before we ran it

A localizer cannot return a file it never indexed. Last time we only checked that after reading the misses, which is the wrong order: a check run afterwards can only be used to excuse them. So this time the check ran first, and its result is committed next to the frozen sample, before any case was scored. In 105 of the 108 scored cases the answer is somewhere in the index. In 3 it is not:

• filamentphp/filament#20371 package.json
• TryGhost/Ghost#27797 pnpm-workspace.yaml
• TryGhost/Ghost#29612 ghost/core/core/server/services/email-rendering/partials/email-wrapper.hbs, ghost/core/core/server/services/email-service/email-templates/partials/styles.hbs

They are scored as misses anyway, as they were in both earlier runs. Dropping them would give 58.1% (61 of 105), and that is a diagnostic we use to decide what to index next, not a result. A customer whose bug lives in a file we do not read gets a wrong answer from us, and our reason for it is not their problem.

What moved, including what moved the wrong way

This run shares no case with the previous one, so there is no like-for-like case comparison to show you. What there is instead is spread. Across 8 repositories the best scores 10 of 12 and the worst 5 of 15, on paperless-ngx/paperless-ngx. An aggregate that hides a collapsed repository is not a summary, so the per-repository split is printed under the table below and in the results file.

Repository routing is 102 of 108. Eight repositories share one workspace here, two more than last time, so picking the right repository before picking a file is a harder question than it was and the number went the right way anyway.

Twelve of the 120 selected cases are not scored. Six because the merged fix creates a file that cannot exist in the tree we indexed, and six because extraction read the issue as several bugs or none, which is a sampled model call and varies between runs. Counting all twelve as misses gives 50.8% (61 of 120), which is on this page for exactly that reason.

Every case, including the ones we got wrong

All 108 scored cases. Each row links to the public issue, shows the file Watari ranked first, and shows the file the maintainer actually changed. Per repository: element-web 7 of 13 · paperless-ngx 5 of 15 · gitea 7 of 14 · jellyfin 9 of 14 · dolphinscheduler 10 of 14 · filament 8 of 15 · Ghost 5 of 11 · langflow 10 of 12.

Feature requests, measured the same way

Watari also maps small, specific feature requests, the kind that name a change to code that already exists, and drafts a pull request for them. So we measured that too, under its own protocol written before the sample was chosen: 40 issues from 4 of the same repositories, each labelled a feature by its own maintainers, scored against the files their merged pull request changed.

On the 28 Watari judged specific enough to map, the top-ranked file was one the maintainers changed 67.9% of the time (19 of 28, 95% interval 49.3% to 82.1%), and one appeared in the top five 85.7% of the time (24 of 28). Counting every excluded or declined case as a miss gives 47.5% (19 of 40).

That is above the bug figure, and the intervals overlap, so the fair reading is that feature requests are located at least as well as bugs, not better. The sample is small and says so. One more thing it showed: 6 of the 40 were read by extraction as bugs, not feature requests. In the product those are handled as bugs, and if you disagree with that call, the Mismapped credit covers it.

results.v5.json every case, committed unedited, after sample.v5.frozen.json.

What this does not prove

This measures one step of six: finding the file. It says nothing about whether the draft pull request we write is any good, whether the tests we add are meaningful, or whether the customer-facing RCA reads well. Those can be shown and timestamped, and they are, on the walkthrough page. They cannot be scored against ground truth a third party already produced, so we do not put a number on them.

How to read 56.5% (61 of 108)

Localizing a bug from a report written by a user is much harder than from an issue written by a developer with a stack trace attached. On the two published academic benchmarks built from user-written reports, the best reported top-1 file accuracy is 41.6% on LinuxFLBench (250 real user-submitted bug reports) and 28.5% on MULocBench (1,100 issues across 46 projects). Both are quoted from those papers. They are the two figures on this page we did not measure.

One difference runs in our favour and should be said out loud: both of those hand the model the repository. We searched 8 repositories at once and had to pick the right one first, which is the routing figure above.