We wrote down what we were going to measure, then measured it.

This run shares no case with the previous one, so there is no like-for-like case comparison to show you. What there is instead is spread. Across 7 repositories the best scores 13 of 13 and the worst 10 of 14, on jupyterlab/jupyterlab. An aggregate that hides a weak repository is not a summary, so the per-repository split is printed under the table below and in the results file.

Repository routing is 90 of 92. 7 repositories share one workspace here, so the search has to pick the right repository before it can pick a file.

13 of the 105 selected cases are not scored: 1 because the merged fix creates a file that cannot exist in the tree we indexed, and 12 because extraction read the issue as several bugs or none, which is a sampled model call and varies between runs. Counting all 13 as misses gives 76.2% (80 of 105), which is on this page for exactly that reason.

All 92 scored cases. Each row links to the public issue, shows the file Watari ranked first, and shows the file the maintainer actually changed. Per repository: jupyterlab 10 of 14 · streamlit 14 of 15 · hugo 11 of 13 · OpenRefine 9 of 12 · atuin 11 of 13 · gradio 12 of 12 · k6 13 of 13.

This measures one step of six: finding the file. It says nothing about whether the draft pull request we write is any good, whether the tests we add are meaningful, or whether the customer-facing RCA reads well. Those can be shown and timestamped, and they are, on the walkthrough page. They cannot be scored against ground truth a third party already produced, so we do not put a number on them.

Version 1 of this protocol measured 47.1% (8 of 17) with a 95% interval of 26.2% to 69.0%. Version 2 measured 58.9% (33 of 56), 45.9% to 70.8%. Version 3 measured 64.3% (36 of 56) on the same cases as version 2 against a fuller index, and was published as a ceiling because the change it measured had been chosen by looking at those cases. Version 4 measured 56.5% (61 of 108), 47.1% to 65.5%, on repositories nobody here had read, and was the standing figure until version 6 replaced it. Version 6 measured 62.6% (67 of 107), 53.2% to 71.2%, on the product exactly as deployed, and was the standing figure until the run above replaced it. None of the five is revised or withdrawn, and all their results files sit in the same public repository. A wider sample was the point of the second version, a fuller index the point of the third, repositories nobody here had read the point of the fourth, the product exactly as deployed the point of the sixth, and the product as deployed today, on repositories new again, the point of this one.