ScaffBench 2.2: one number, three trials, no excuses

What this bench is

ScaffBench asks one question: can a coding agent turn a prompt into an entire full-stack project that actually works? Not a snippet, not a patch to an existing repo — the whole thing: manifests, configs, wiring, code, across eight ecosystems (TypeScript, Rust, Go, Python, Java, .NET, Elixir, React Native, and polyglot combinations).

The agent hands back a folder. We grade the folder. Nothing the agent claims counts — only what builds.

What we run

13 specs × 3 trials × each model. Every spec is a realistic project brief — the stack, the required libraries, the features — for things like an AI search workbench (Hono + oRPC + Drizzle + Qdrant + OpenSearch), a Spring Boot + jOOQ + Keycloak API, a Go realtime service, a gRPC contract shared by a TypeScript web app and a Go service, or a pure-engineering "frontier" spec no scaffolder could produce.

The prompt rules are short: create exactly one project directory, don't ask questions, don't start dev servers — and, new in 2.2, you may install dependencies, query package registries, and run builds to verify your work before finishing. Earlier versions were closed-book; models kept inventing dependency versions that don't exist. Now checking is allowed. Not checking is a choice.

Each spec runs three times, independently — fresh workspace, no memory between attempts, spec order shuffled per round. Models don't produce the same project twice, so one attempt tells you what happened; three tell you what's true. A spec scored 2/3 isn't a pass or a fail — it's a 67%.

How we check results

Every generated project is validated cold, on a clean machine, the same way for every model:

1. Install — real package managers against live registries. Invented versions die here.
2. Build + typecheck — the ecosystem's real toolchain (cargo check --workspace, dotnet build on the solution, tsc, go build ./..., Expo export, protobuf codegen…).
3. Quality gates — lint, format, tests. All of it, every run.

The published number is the Full pass rate: the share of trials that clear everything. A project that compiles but fails its own lint or tests doesn't fully pass — that's the difference between "builds" and "done."

The harness also refuses to blame models for our problems: provider rate-limit deaths, missing toolchains, and network flakes are classified as infrastructure and excluded — automatically, with the evidence recorded. And rows whose scores are within single-cohort sampling noise share a rank on the board instead of pretending an order exists.

How this compares to other benchmarks

ScaffBench is one-shot, not iterate-to-green. The agent gets a spec and a full shell; what it hands back is final — no grading inside its own sandbox, no retries against the verdict, bounded only by a wall-clock ceiling and a budget cap. SWE-bench and Terminal-Bench grade iteration inside the model's own environment; Aider's polyglot suite allows bounded retries against a test suite; HumanEval-style suites never execute in a real environment at all. ScaffBench grades the artifact, after the fact, under identical conditions — which is why a model can top agentic leaderboards and still lose here to a smaller sibling that verifies before it pins.

Two consequences of that design: failures are choices, not handcuffs — a run that dies on a dependency version that doesn't exist chose not to look it up — and the assisted lanes (where our MCP scaffolder drives) measure something different from raw capability: restraint, because with a working project handed to it, what separates models is whether their edits keep it working.

First 2.2 cohort: the GPT-5.6 family at high effort

Model	Full pass	Builds	Index
GPT-5.6 Luna	44% (17/39)	72%	78
GPT-5.6 Sol	41% (16/39)	67%	74
GPT-5.6 Terra	38% (15/39)	67%	74

Three things worth knowing behind those numbers:

• The smallest model ties the flagship. The gap between Luna and Sol is two cells out of 39 — statistical noise — and that's the finding: scaffolding rewards restraint and convention, and Sol's extra capability doesn't convert. It does show somewhere: Sol swept ai-search-workbench 3/3, the hardest TypeScript spec, which no model had ever passed. It gave those cells back by breaking its own build scripts on the polyglot gRPC spec that Luna passed 3/3.
• Self-verification works. The Keycloak spec — failed by every previous GPT run on hallucinated dependency versions — now passes for all three models. Given permission to check the registry, they check.
• The Full tier bites. All three models build roughly two-thirds of their projects; barely half of those survive lint, format, and tests. That drop is the space between "it compiles" and "you'd keep it."

Honest notes

• Three trials beat one, but 39 samples still carry ±8pp of noise — treat close rows as tied, because the board does.
• These three rows are one vendor family. Claude and the free-tier endpoints run next; nothing from older harness versions is mixed into this board.
• Every 2.2 row is ranked-eligible: same harness (2.2.1), same prompt, same validator, three consistent trials per cell. Older boards stay published as history, clearly marked.

The bench runs in the open — specs, harness, validator, and every number's provenance.