Can a Harness Make a Small Model Match a Large One?

Is a good harness enough to make a small model match a large one? By harness, take the definition from the literature that studies it: the software layer between the model and the task, which constructs the context the model sees, mediates its tool calls, validates its outputs, and decides when to retry, escalate, or stop [3]. Scaffolding, adapter, agent framework, sampling policy — they are all this one thing, and everything below treats them as one thing.

The question sounds like it is about models. It is really about where difficulty lives. When a task is hard, it is hard either in the single act of thinking — the moment the model must produce something it may simply be unable to produce — or in everything around that moment: finding the right file, running the code, noticing the error, trying again. If most of the difficulty is in the surroundings, then better surroundings should let a smaller model do the job, and model size stops being a ceiling and becomes a purchasing decision.

The line the evidence draws runs between tasks, not between models, and it turns on three properties of the task: whether it affords a check independent of the model's own judgement, whether its problems sit in the band the small model already sometimes reaches, and whether its horizon is short. Two sections come first, on how large the harness lever is and who it favours; the three that establish these properties follow, then what substitution costs and how to check the next parity claim you are shown.

The lever is larger than you think

Start with the part that is not in dispute. Hold the model still, change only the scaffolding around it, and scores move by amounts that dwarf a typical model upgrade.

On the Terminal-Bench 2.0 leaderboard, Claude Opus 4.6 scores 58.0%±2.9 under Claude Code, 62.9%±2.7 under Terminus 2, and 76.4%±2.4 under Meta-Harness — one model, three harnesses, three separate submissions by three different parties [1]. A benchmark built to isolate the effect under controlled conditions gets a sharper result: with the same GLM 5.1 backbone, a minimal adapter that asks the model to emit a unified diff directly scores 19.1% Pass@1, while a full adapter that lets the model edit files and exports the patch from Git state reaches 73.4% [2]. The diagnosis is in the failure column. Under the minimal adapter 69.1% of patches failed to apply at all; under the full adapter, 1.5% [2]. The bottleneck was not that the model could not edit code, but "the fragility of directly generating unified-diff text: line numbers, context, hunk headers, or trailing newlines can make the patch fail to apply" [2].

That is what a harness supplies at its most basic: not intelligence, but a way for correct work to survive being handed in. And the effect is large enough to swamp model differences outright. One analysis, reading a public leaderboard, found that on a standardized scaffold six leading frontier models spanned only 4.9 percentage points on SWE-bench Pro (41.0% to 45.9%), while holding one of those models fixed and varying only the harness opened a 9.5-point gap [3].

So if you are choosing between spending a quarter on your scaffolding and spending it on a model upgrade, the scaffolding is often the better trade. That is a genuine and useful finding. It is not, however, the finding people reach for it to support.

The lever is not symmetric — but not in the way you expect

Here it pays to separate two things that the word "lever" runs together: how much a model's score varies across harnesses, and how much one good harness gains it. The evidence answers these differently, and conflating them is where most parity claims go wrong.

Variance first. In the matched comparisons available here, it is larger for the weaker model. The five-harness sweep above found a 12.5-point spread on the stronger GLM 5.1 and a 27.4-point spread on the smaller Qwen 3.6-flash [2]. A factorial study across a capability-spanning pool of backends reports the same gradient: "stronger model backends tend to achieve higher mean scores while exhibiting lower cross-harness variance", suggesting stronger models may be more tolerant of differences in prompting and tooling [4]. On that evidence, weaker models tend to be more at the mercy of their scaffolding than stronger ones.

This is what makes an unmatched comparison uninterpretable, and the field mostly runs unmatched: the harness "is rarely disclosed and almost never held constant across comparisons" [3]. When the scaffold sits on one side only, the number you are shown is dominated by the axis with the larger spread, which is the small model's. Show me a scaffolded small model beating an unscaffolded large one and you have measured the harness, not the model.

But gain is a different measurement, and it points different ways.

On a hard reasoning benchmark, one frontier model gained more than the small model and the other gained less. Qwen3.6-35B's own first-shot accuracy on that benchmark is 19.00 against Claude Sonnet 4.6's 48.00, so the full stack running entirely on Qwen finishes at 24.00 — below Sonnet's single-shot score [5]. On a coding benchmark the same small model gained more in absolute terms, 47.43 to 59.43, the largest gain the study reports, and still landed below the frontier model's unscaffolded 77.14 [5].

So the honest statement is regime-dependent. Harness variance favours the weak model, which is why unmatched comparisons flatter it. A specific good scaffold's gain does not reliably favour anybody — it depends on the scaffold and the task — and in the matched cases available it has not been enough to close an absolute gap.

The starkest version is a benchmark of long-horizon, multi-file engineering tasks run through one unified SWE-Agent scaffold. On its public set: GPT-5 at 23.3%, Claude Opus 4.1 at 22.7%, SWE-Smith-32B at 6.8%, Qwen-3 32B at 3.4% [6]. Same harness on both sides, and the ordering does not budge.

Where parity is real

And yet parity results exist, and they share one property — the first of the three that decide substitution: a checker. A graded answer, a passing test suite, a well-formed tool call — something that lets the system tell a good attempt from a bad one with no human in the loop, so that extra attempts convert mechanically into accuracy.

Four cases carry the literature, and three of them run on test-time compute — spending the budget on attempts and checks at answer time rather than on a bigger model. On graded mathematics, compute-optimal test-time scaling puts Llama-3.2-3B-Instruct at 75.6% on MATH-500 and 30.0% on AIME24 against an unscaffolded chain-of-thought Llama-3.1-405B-Instruct's 71.4% and 23.3%, across a 135× parameter gap [7]. A FLOPs-matched study finds test-time compute beating a 14× larger model, on the stated precondition that it applies "on problems where a smaller base model attains somewhat non-trivial success rates" [8].

On unit-tested code, repeated sampling lifts DeepSeek-Coder-V2-Instruct on SWE-bench Lite from 15.9% at one sample to 56% at 250 [9]. And on structured tool calls, the best open-weight model tested is statistically equivalent to GPT-5 across 30 tasks, at a pre-registered equivalence margin of ±10 percentage points [10].

Note that the last of those multiplies no attempts at all. It runs at temperature 0, one run per observation, with a byte-identical system prompt and a single runner for every model [10]. There the checker makes the benchmark scorable; it is not what makes the small model competitive. A checker can do either of two jobs — make a task scorable at all, or turn extra attempts into accuracy — and only the second is "buy more tries."

What the average was hiding

The checker gets you parity on the mix of problems a benchmark happens to contain. Split that mix by difficulty and the headline dies.

One evaluator runs all 88 of its SWE-bench Verified models through a single bash-only harness, giving every model one tool and the same configuration so that, in its words, "This puts the evaluation burden squarely on the model rather than the harness" [11]. It then publishes resolution rate per difficulty band, with 194 tasks under 15 minutes, 261 from 15 minutes to an hour, 42 from one to four hours, and 3 above four hours [11].

The easy band compresses the field [11]. The one-to-four-hour band pulls it apart.

Gaps of single digits become gaps of tens of points, under an identical harness — and since the two easy bands hold 194 and 261 tasks against 42 and 3 in the two hard ones, the aggregate is dominated almost entirely by the region where everyone converges [11]. The board itself now carries an "Archived Benchmark" banner explaining that "performance on this benchmark has saturated, we no longer run this benchmark on new model releases" [11], which is the same point from the other end.

The reversal shows up everywhere it is looked for. Push Llama-3.2-1B-Instruct's budget to N=512 and it beats that same unscaffolded 405B baseline on MATH-500, 72.2% against 71.4%, while losing to it on AIME24, 10.0% against 23.3%; the paper states plainly that test-time scaling "is more effective on simpler tasks than on more complex tasks" [7]. The FLOPs-matched study finds pretraining more effective on the hardest bins and concludes the two compute types "may not be 1-to-1 exchangeable" [8].

A parity number quoted as one aggregate is usually a claim about task mix — which makes the difficulty band the second deciding property.

What the harness does not supply

Ask what is left once the harness has done everything it can. Three things, but only one of them is new: the other two are deciding properties already met, seen now from the scaffolding's side — the checker, and the third of them, horizon.

The new one is selection, and it is less final than it looks. When you sample repeatedly, coverage — the chance some attempt is right — keeps climbing: on MATH with Llama-3-8B-Instruct it goes from 82.9% at 100 samples to 98.44% at 10,000 [9]. But majority voting and reward models move the actually-selectable score only from 40.50% to 41.41% over the same range [9]. The right answer is in the pile and nothing can pick it out. That said, the ceiling is partly an artefact of weak selectors: ensembling many weak verifiers under weak supervision improves over Pass@1 by 17.9% for 8B models and 14.5% for 70B, "only via increased sampling at test time rather than parameter tuning" [12]. Selection is an engineering frontier, not a wall.

The checker returns as a demand for independence. The operative property of a good checker is not that it is cheap but that it does not route through the model's own judgement. Where it does, it is the weakest check available: the prompting-based self-correction baseline "deteriorates the final predictions, as LMs struggle to identify errors in their reasoning" [13].

Horizon comes last, and it is the one that has yielded least. The tool-use ladder certifies equivalence at its lowest tiers, but on long-horizon planning under persistent constraints the frontier model is strictly superior — and the absolute pass rates there are 10% and 0%, so the paper notes that "neither side reaches practitioner-relevant reliability" on that tier [10]. Under one unified scaffold on long-horizon engineering work the ordering held with a gap of tens of points [6]. Scaffolding does narrow it. A memory scaffold for long-horizon agents brings Claude Sonnet 4.5 from 37.6% to 45.9% on Terminal-Bench 2.0 while Claude Opus 4.6 moves 43.5% to 45.9% — the two converge [14]. But read the setup rather than the table: the memory agent is Claude Opus 4.6 [14]. The configuration that brings the weaker model level with the stronger one is the weaker model plus the stronger one. That is a real engineering result, and it is not substitution.

What it costs, and the cheaper third option

Every parity result above was bought, and the price runs in both directions.

Holding one framework constant, the SWE-bench Lite comparison runs in the open model's favour [9].

A genuine win; the paper's own summary is that the open model "can achieve the same issue solve-rate as closed-source frontier models for under a third of the price" [9]. On HumanEval the arithmetic inverts: no scaffold placed on GPT-3.5 reached zero-shot GPT-4's 89.6% at $1.93, and the most elaborate attempt scored 80.4% for $9.49 [15].

Two accounting caveats matter before you budget from any of this. First, the coding figures are pass@k — an issue counts as solved if any attempt passes, which is an oracle you do not have at selection time, and the same paper reports that 11.3% of SWE-bench Lite problems have flaky test suites that sometimes fail the dataset's own ground-truth patches [9]. Second, more spend is not reliably more accuracy: across 21,730 rollouts on nine models and nine benchmarks, higher reasoning settings gave equal or lower accuracy in 21 of 36 configurations tested [16].

And the same paper that reports the HumanEval inversion also reports the option that beats both framings of this question. Its escalation baseline starts on a cheap model and moves each unsolved problem up a chain of increasing cost — Llama-3-8B, then GPT-3.5, then Llama-3-70B, then GPT-4 — whenever a proposed solution fails one of the example tests. That reached 85.0% for $0.27 against zero-shot GPT-4's 89.6% for $1.93, and in a robustness re-run on older model versions came out ahead of single GPT-4, 87.6% (86.6–88.4) at $0.42 against 86.5% (84.8–87.2) at $2.94 — point estimates apart, intervals overlapping [15]. The paper's blunter finding is that the elaborate architectures it re-evaluated "do not outperform simple baselines" on this benchmark at all [15]. Escalation needs the same thing substitution needs — a check cheap enough to run on every attempt — but it fails safe when the check says no.

Five questions for any parity claim

This is the most portable thing here.

1. Was the same, named harness used on both sides? Public comparisons almost never hold it constant [3], and a five-harness sweep reports a 12.5 pp spread on one model and 27.4 pp on another [2]. Check this first because it is the largest thing you can check.
2. Which difficulty bands does the number cover — and how much of the benchmark's mass sits in the easy ones [11]?
3. How many runs produced it, and does it carry an interval?
4. Is cost reported next to accuracy — and does the accounting bill the checker's own inference?
5. What would count as the right answer? If the only available check is the model's own judgement, you are outside the regime where scaffolding helps: a small model's prompted self-correction degrades its own predictions [13].

You can run the first test yourself. The minimal bash-only reference agent is public, about a hundred lines of Python for the agent class, gives the model no tool but bash, and reports scoring above 74% on SWE-bench Verified [17]. The evaluator quoted above publishes the protocol it runs that agent under: one tool for every model, and each model run with the provider's default configuration except for the token limit, which is set as high as possible [11].

Conclusion

No — a good harness is not sufficient, and asking whether it is "enough" is the wrong shape of question. It is a large lever, frequently larger than a model upgrade. The weak model is the more exposed to it; neither model reliably gains more from it. What it buys is bounded by the task, not by the scaffolding.

Three properties of the task decide it, one established in each of the three middle sections: an independent check, a difficulty band the small model already sometimes reaches, and a short horizon. Selection is not among them; it is the residue that keeps yielding to better engineering. Where all three hold, substitution is real and often cheaper. Where any fails, the matched comparisons available leave the ordering intact and the gap wide.

So test your task, not your model. If it has an executable checker and runs in minutes, scaffold the small model and measure. If it runs for hours, or if the only judge of a good answer is a model's opinion, buy the capability. And if what you actually want is the cost saving rather than the architecture, escalation is the option to price first: it needs the same cheap checker, and it keeps the large model on call instead of betting the task on its absence.

Limitations

The evidence cannot establish the one comparison this question most needs: a study that holds the harness fixed and sweeps backbone scale as the treatment, at matched difficulty, on current-generation models. Every asymmetry reported above is a by-product of a paper measuring something else. The dollar figures are quoted at 2024 API prices and cannot be re-priced, because no source publishes per-trajectory token counts. No accounting anywhere bills the verifier's own inference, so every cost advantage stated here is an upper bound on the real one. And no source measures wall-clock for a scaffolded small-model trajectory against a single large-model call, so whether the price win survives a latency ceiling is unknown.

Several of the most decision-relevant sources are recent preprints and leaderboards with a single origin each, not independently replicated. The per-model, per-difficulty-band decomposition that this piece leans on hardest exists nowhere else, and the board carrying it is the archived one quoted above. The long-horizon benchmark is cited at the version captured here, and its resolve rates should be re-checked against the current revision before they are relied on [6]. Finally, the strongest arithmetic in this piece is prior-generation — 2024 models and benchmarks — and whether those exchange rates transfer to today's frontier is itself unmeasured.