1) Use purpose-built test harnesses We use Claude Code or Codex
2) Test one-shot tasks We work in sessions
Sessions are messy, we start with a large primary task, then some cleanup, an adjacent fix here another over there. We start, stop, and change our minds.
That changes both cost and quality. Cache TTLs expire. Context grows.
I am working on creating one. Here's my rough plan: 1) Use Claude Code and Codex 2) Use session shaped workloads. Stitch multiple SWE bench verified tasks into one big session. 2.a) Use tasks from the same repo to ensure topical continuity. 3) Headline metrics: dollar cost vs quality. 3.a) Secondary metrics: turn count, time to completion
Open Questions a) Does this problem make sense? b) Does my benchmark spec make sense? c) Are 10 SWE bench verified task starting with a hard one the right workload shape?
At a high level, it
- Mines tasks from your merged PRs/commits - Replays them in Docker containers with different harness settings (change model / reasoning effort / AGENTS.md / etc) - Grades the patches on various attributes (tests, equivalence with human patch, code quality)
The goal is to get a sense of how agents perform on your tasks, with your context, using the tools you do.
This is currently one-shot but I'd definitely like to explore session-based benchmarks as well. There are some interesting papers that just came out on this https://arxiv.org/abs/2606.29957 https://arxiv.org/abs/2606.30573