some headlines
- 3000 rubrics on code quality. First benchmark to measure: "would this code get actually merged?"
- 20+ expert open-source maintainer created tasks on their own repos to capture their opinion & taste.
- 40+ hours of real human work per task. total 1000+ hours of real life software maintainer work captured in dataset
- results in 81% lower false positive rate than SWE-Bench Pro
- High quality bar: many QA stages & each task manually reviewed by Cognition researchers (examples in post)
Opus 4.8 scores 13% on FrontierCode Diamond.
one of my goals was also to datamine interesting stuff even on the easy tasks. for example, if you squint you can see the answer to "WTF Happened in late 2025" with coding models: https://x.com/swyx/status/2064081945567580323
i think something like ~60% llm as judge rubrics and the rest as described. every rubric validated by maintainer. 3000 rubrics