By Model VS · Guide text updated . Counts and the first rows are filled from the saved specialist snapshot at build time. Model VS does not re-run SWE-bench. A linked model card is the evidence the source row points at, not a reproduction by us. Report a correction if a flag or link does not match the file.
Verified describes the tasks, not every score.
SWE-bench Verified is a human-filtered subset of 500 instances from SWE-bench, created in collaboration with OpenAI. Human annotators reviewed each instance to ensure the problem descriptions are clear, the test patches are correct, and the tasks are solvable given the available information.
That is a statement about the questions, not about a vendor’s press release. The subset was filtered so the issue text and the tests were considered usable by human reviewers. OpenAI’s introduction of the subset describes the same idea: annotators screened samples for underspecified issues and tests that reject valid fixes, and the released slice has 500 samples. Read that note for the annotation scale and the limits they already named. Static datasets go stale. OpenAI has since argued, in a later note, that contamination and remaining test flaws are reasons they stopped treating this subset as a frontier score. Whether you agree with that recommendation or not, it is a reason to cite the date and the harness, not a naked percentage.
Other slices exist. SWE-bench Lite is a different, smaller set. A full-system leaderboard and a bash-only leaderboard are different comparisons even when both say Verified. If two blog posts quote “SWE-bench” without the subset and the harness, they may not be comparable. Our table is labeled SWE-bench Verified because that is the benchmark id in the file. The label does not upgrade every row into a checked submission.
Our snapshot has a second, narrower field.
Each specialist row carries a boolean verified field from the leaderboard record we sync, plus a link that is supposed to be the submission evidence. How we rate already says a benchmark name is not a submission certificate. This page is the worked case. “Verified” in the title means the task subset. “Source marks this submission verified” would mean the row’s own flag is true. “Community submission — not marked verified” means the flag is false. Those sentences are not synonyms.
A false flag does not mean the number is fabricated, and a true flag would not mean Model VS reproduced the run. It means the upstream record did or did not mark the submission as verified. You still have to open the link and look for the model id, the scaffold, the test split, the attempt count, and the inference budget. If the link is a generic model card with a single percentage and no harness, write “unknown” next to those fields. Unknown is a result. It is how you avoid promoting a screenshot.
The file also has a dataset timestamp that can be older than the day we copied the leaderboard. Both times are printed in the generated section. Quote the one you mean. “As of our specialist sync” and “as of the dataset card” can be different days. Neither is “this morning” unless the timestamp says so.
Count the flags before you quote the top of the list.
The build reports how many saved rows are marked verified, how many are not, and how many have no flag. It then prints the first five rows in file order so you can see the flag sitting next to a high reported score. A high score with an unverified flag is still a high reported score. It is not a certificate. Do not drop the flag when you move the number into a slide.
Snapshot saved October 6, 2026 (2026-10-06T09:00:11.687Z). SWE-bench Verified dataset timestamp on that file: August 16, 2026. Rows in the saved file: 67. Marked verified: 0. Not marked verified: 67. Verification field absent: 0.
| Rank | Model id | Reported score | Submission flag in our snapshot | Linked evidence |
|---|---|---|---|---|
| 1 | ornith-ai/Ornith-1.5-397B | 86.00 % resolved | Community submission — not marked verified | Ornith-1.5-397B model card |
| 2 | mindlab-research/Macaron-V1-Venti | 85.60 % resolved | Community submission — not marked verified | Model Card |
| 3 | mindlab-research/Macaron-V1-Coding-Venti | 85.60 % resolved | Community submission — not marked verified | Model Card |
| 4 | ornith-ai/Ornith-1.0-397B | 82.40 % resolved | Community submission — not marked verified | Ornith-1.0-397B model card |
| 5 | deepseek-ai/DeepSeek-V4-Pro | 80.60 % resolved | Community submission — not marked verified | Model Card |
The same specialist file's GPQA table, dataset timestamp September 21, 2026, has 100 rows: 0 marked verified, 100 not marked verified, 0 with no flag. The flag question is not unique to the coding benchmark.
The same file’s GPQA table is included in that count when it exists, because the habit is not limited to coding. A reasoning accuracy with an unverified flag has the same citation problem. GPQA and SWE-bench still must not be added together. They do not share a unit.
The thing that posted the score may include tools.
SWE-bench scores are produced by a system that edits a repository and runs tests. That system can include a scaffold, a retry loop, a test viewer, retrieval, or a specific agent harness. The SWE-bench site itself distinguishes a minimal bash setup from arbitrary systems. A percentage copied from a full-system board is not the percentage you will get from a chat box with no tools, and it is not the percentage you will get from your IDE unless your IDE is that system.
Ask five questions before the number enters a procurement note:
- Which subset: Verified, Lite, or another split, and how many instances does that page claim?
- Which harness: bash-only, a named agent, or an unnamed internal runner?
- Which model id, including size and reasoning setting, not the product family?
- Is the submission flag true, false, or missing in the snapshot you are citing, and what does the linked page actually show?
- What is the date of the source page and the date of our file?
If any answer is missing, the honest shortlist line is “reported percent, harness unknown,” not “best coding model.” Our specialist view says the same thing in shorter form: agent and coding results may include a scaffold. This essay is the longer version of that sentence.
LiveBench coding will not confirm this table.
LiveBench’s Coding category, on the release in our other file, is an average of its own task columns. It does not resolve GitHub issues in the SWE-bench repositories, and it does not use the Verified subset. A model can sit high on one and off the saved list of the other. That is not a sync error. It is two exams. Use the category guide when you want those columns, and use this page when you want the submission list.
Arena preference is a third exam. People voting on chat replies are not grading patches. Do not break a tie on SWE-bench by looking at who has the friendlier Arena rank. If you need both a conversational assistant and a coding agent, you may end up with two systems. That is allowed. The decision protocol tells you to label a comparison of configured systems as a comparison of systems.
A usable citation is longer than the percentage.
A citation that survives a week later has the model id, the subset, the harness if known, the reported metric, the submission flag, the link, and both timestamps. Example shape, with the numbers left for the table above to supply: “On the Model VS specialist snapshot saved on the printed sync time, the first SWE-bench Verified row reported the printed percent resolved. The submission flag in that file was the printed flag. Evidence link: the printed URL. Dataset timestamp: the printed dataset time. Model VS did not reproduce the run.”
Then run a tiny version of the job on a repository you are allowed to touch. One issue with a real test is more informative than moving a public percentage into a roadmap. Keep the patch, the command, and the fail log. If the public row’s harness is not your harness, say that the pilot is a new experiment. The public number remains context. It does not become your pass rate.
We will not mark a row verified because the benchmark title contains the word. The flag in the file is the flag we show. If the upstream record changes, the next validated snapshot can change the count. Until that file changes, the count on this page is the count in the file.