Adding scaffolding around an AI model can move performance more than swapping the model. It can also degrade it. Here is what the public data actually says, and how to read it without getting fooled.
Industry analysis · ~12 min · 30+ cited sources
The frame
The people shipping the harnesses are the first to say when not to use one
Two of the most consequential agentic-AI labs in the world publish, in the same calendar year, what reads like a contradiction. They are not contradicting each other. They are bounding the claim.
“Agentic systems often trade latency and cost for better task performance, and you should consider when this tradeoff makes sense.”
This guide reads the harness question through the data, not the marketing. The harness can be the bigger lever than the model. It can also be the bigger anchor. Both are documented; neither is universal. The honest job is figuring out which side of the line a given task sits on, and refusing to pretend the literature says more than it does.
Definitions
What a harness actually is
Anthropic’s cleanest first-party definition: “the basic building block of agentic systems is an LLM enhanced with augmentations such as retrieval, tools, and memory.” [1] Simon Willison’s converged operational form is tighter still: “an LLM agent runs tools in a loop to achieve a goal.” [3] OpenAI frames an agent as a system that “independently accomplishes tasks on your behalf,” and explicitly recommends starting with a single agent and evolving to multi-agent only when needed. [4]
Under these definitions, a harness is everything sitting between the raw model API call and the eventual answer. Six primitives recur across the literature.
01
System prompt
Role, rules, response shape
02
Tools
Functions the model can call
03
Memory
Retrieval + persistent state
04
Planner
Scaffolded reasoning / search
05
Orchestrator
Multi-agent role assignment
06
Evaluator loop
Self-critique, retry, score
Each primitive has at least one foundational paper behind it. Reasoning-plus-tool-use as a loop comes from ReAct [5]. Self-reflection from Reflexion [6]. Search-over-thoughts from Tree of Thoughts [7]. Self-supervised tool calling from Toolformer [8]. Role-based multi-agent from MetaGPT [9]. The autonomous task-loop archetype from AutoGPT and BabyAGI [10]. Add or subtract these in any combination and you get a different harness; a fair benchmark holds the model fixed and varies only this stack.
The killer datum
Same base model, same benchmark, 9.5× performance
The cleanest single-citation proof in the literature comes from the SWE-agent paper (Yang, Jimenez et al., NeurIPS 2024). Same model. Same 2,294 SWE-bench issues. Only the harness changes.
Read the columns left to right. The model is constant. The benchmark is constant. The harness goes from BM25 retrieval handed to a single API call to a bash loop to a purpose-built Agent-Computer Interface. Performance climbs from 1.31% to 12.47% on the full set: a 9.52× relative lift, +11.16 percentage points, entirely attributable to scaffolding. On SWE-bench Lite the ACI alone (over the shell-only intermediate) adds another +64% relative on top of any-agent-loop-at-all. [11]
This is the rare benchmark row where the harness contribution is isolable. The same group ran the comparison; the same paper publishes the ablation; the model and dataset are identical on both sides of the arrow. Every other “harness lift” number in the field should be read against this one as the gold standard for what cleanly-controlled looks like.
The thesis benchmark
GAIA tells the harness-as-lever story most cleanly
GAIA (General AI Assistants, Mialon et al., 2023) tests real-world assistant tasks across reasoning, multi-step planning, tool use, web browse, and file handling. Three properties make it the best benchmark for this conversation: a clean human ceiling, a clean no-harness baseline, and a tracked progression through identifiable harness generations.
Sources for every bar live in the index [12][13][14]. The trajectory is 15% to 92.36% in roughly thirty months, on a benchmark whose human baseline is 92%. Part of the lift is model upgrades. The cleanly attributable harness contribution is recoverable from same-model comparisons inside the progression: GPT-4o no-harness at 32% versus GPT-4o-class models inside Langfun or h2oGPTe at 49–65%, a +17 to +33 percentage-point swing from scaffolding alone. [12]
Supporting evidence
Same lift signal shows up across agentic benchmarks
Three other benchmarks show the same shape: low single-figure baseline under default agent, large climb as harnesses mature. Each carries its own caveat, included where the primary sources flag it.
Steel.dev’s own leaderboard registry warns that “judge methodology varies significantly across submissions - human eval, WebJudge, and custom agentic judges produce different scores for the same agent.” Browser-agent figures above 90% on heterogeneous-judge boards are not directly comparable. [18]
Independent supporting case from a different lineage: Tree of Thoughts on Game of 24. GPT-4 with chain-of-thought scored 4%; the same model wrapped in a search-over-thoughts planner scored 74% in the original paper, with the authors’ own reproduction landing at 69% (−5 percentage points versus the paper, openly disclosed as decoding stochasticity). [7] Same base model, +65 to +70 percentage points from the planner alone.
Scaffold-sensitivity warning
Anthropic’s own Claude Opus 4.6 announcement publishes both 49.3% AND 81.42% on SWE-bench Verified for the same model, averaged over 25 trials, separated only by “a prompt modification.” That is a ~32 percentage-point gap from prompt alone, on the second-most-quoted coding benchmark in the field. Claude 3.7’s headline 70.3% was the vanilla 63.7% plus parallel sampling, rejection scoring, and visible-regression-test re-rank; Anthropic disclosed the stack on their own page. Any “the SWE-bench Verified score” of any model is a scaffold-conditioned quantity, not a model-intrinsic capability.[20][21]
The counterweight
When the harness hurts
The literature on harness lift is matched, paper for paper, by literature on harness drag. Nine cited findings cover the field: long-context degradation, deliberation-vulnerable tasks, multi-agent fragility, and the math of compounding error.
Lost in the Middle
Liu et al. · TACL 2024
“Performance can degrade significantly when changing the position of relevant information... performance is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts, even for explicitly long-context models.” A harness that stuffs context measurably hurts retrieval of what sits in the middle. [22]
Context Rot
Chroma · 2025
Tested across 18 frontier models including GPT-4.1, Claude Opus 4, Gemini 2.5 Pro, Qwen3-235B. “Models do not use their context uniformly; instead, their performance grows increasingly unreliable as input length grows.” LongMemEval shows significant gaps between focused (~300 tokens) and full (~113k tokens) prompts. [23]
CoT helps mostly on math
Sprague et al. · ICLR 2025
Meta-analysis of 100+ chain-of-thought papers, 20 datasets, 14 models. “CoT gives strong performance benefits primarily on tasks involving math or logic, with much smaller gains on other types of tasks.” On MMLU, ~95% of CoT’s gain is attributable to questions containing an equals sign. Outside math/symbolic, the scaffolding is roughly inert. [24]
Mind Your Step
Liu, Kambhampati et al. · ICML 2025
On three of six cognitive-psychology-derived tasks, chain-of-thought drops accuracy by up to 36.3 percentage points absolute (OpenAI o1-preview vs GPT-4o). The scaffold “drastically harms LLM/LMM performance in cases where overthinking hurts human performance and humans and models face similar constraints.” [25]
Multi-agent debate fails to beat single-agent
Zhang et al. · arxiv 2025-02
Systematic evaluation of 5 multi-agent-debate methods × 9 benchmarks × 4 models. “MAD often fail to outperform simple single-agent baselines such as Chain-of-Thought and Self-Consistency, even when consuming significantly more inference-time computation.” [26]
Single-agent ≥ multi-agent under equal budget
Tran & Kiela · arxiv 2026-04
Almost all multi-agent benchmarks compare a single agent against a multi-agent system using significantly more compute. Once thinking-token budgets are held constant, single agents match or beat multi-agent systems on multi-hop reasoning. The lift was the budget, not the architecture. [27]
Don’t Build Multi-Agents
Walden Yan · Cognition 2025-06
From the lab that builds Devin: multi-agent architectures produce “fragile systems” because “decision-making ends up being too dispersed and context isn’t able to be shared thoroughly enough.” Parallel subagents risk “conflicting responses, resulting in the reliability issues.” Principle: share context, and share full agent traces, not just individual messages.[2]
Anthropic’s own 15× token tax
Anthropic · 2025-06
Anthropic’s multi-agent research system beat a single agent by +90.2% on their internal eval, at a cost of ~15× more tokens than chat (vs ~4× for single agents). Their own anti-pattern list: don’t use multi-agent for “domains requiring all agents to share identical context, tasks with heavy interdependencies, most coding work, or scenarios demanding real-time agent coordination.” [28]
Compounding error (Husain)
Hamel Husain · 2024
The practitioner math: a 20-step workflow with 95% per-step reliability succeeds 36% of the time end-to-end (0.9520 ≈ 0.358). Any harness that adds steps without proportionally raising per-step reliability degrades overall accuracy. [29]
The harness is model-conditional
Yang et al. · arxiv 2023-06
Rigorous study of AutoGPT on ALFWorld and WebShop. With GPT-4 the loop beats imitation-learning baselines (0.485 vs 0.306 on ALFWorld). With Claude (0.082) and GPT-3.5 (0.075), the same AutoGPT loop actively underperforms IL baselines. The autonomy hype did not survive rigorous benchmarking unless the base model was strong enough. [30]
Most visceral single number · Mind Your Step (ICML 2025)
GPT-4o, no CoT→o1-preview, with CoT
−36.3 percentage points absolute on overthinking-vulnerable tasks [25]
When the benchmark is broken
OpenAI publicly dropped SWE-bench Verified
On 2026-02-23, OpenAI announced they would no longer report SWE-bench Verified scores. The reasons matter more than the announcement: the field had built its narrative on a benchmark with documented contamination and structural test-spec flaws.
OpenAI cited two findings. First, all frontier models tested (including GPT-5.2, Claude Opus 4.5, Gemini 3 Flash) can reproduce verbatim gold patches from the task ID alone; one GPT-5.2 instance solved a supposedly-unsolvable problem by recalling unspecified test requirements that were visible in its own chain of thought. Second, more than 60% of remaining problems have critical flaws in test specification - 49 too narrow, 26 too wide (request features not in the problem statement). The successor benchmark is SWE-bench Pro, built by Scale. [31]
Independent quantification appeared two months earlier: “Does SWE-Bench-Verified Test Agent Ability or Model Memory?” (arxiv 2512.10218) reports frontier models identify the buggy file from issue text alone at 76% accuracy on Verified vs 53% on out-of-distribution repos, and 5-gram verbatim overlap with training data is 35% on Verified vs 18% elsewhere. [32]
Companion case: Devin’s 13.86%
Cognition’s March 2024 headline of 13.86% on SWE-bench was evaluated on a random 25% subset (570 of 2,294 issues), with a 45-minute per-task budget. The “assisted” baselines it was compared against were file-pointed; Cognition’s own technical report acknowledged “neither unassisted nor assisted is strictly comparable to the agent setting.” No formal retraction was issued; the marketing posture quietly shifted. The number was not wrong; the framing was. [33]
The lesson is broader than these two cases. The second-most-quoted coding benchmark of the era is materially compromised, and the most-quoted agent claim of 2024 sat on a 25% subset. Any reasoning about “harness lift” that begins with a leaderboard number inherits whatever flaws that leaderboard carries, including ones the publishing lab itself has since renounced.
Method
How to read a harness number honestly
Distilled from the previous five sections. Eight questions to put to any harness-lift claim before letting it shape a decision.
01
Is the base model the same on both sides of the arrow?
If only the SWE-agent comparison in the literature qualifies, it is because almost no other figure holds the model genuinely constant.
02
Are ablations published?
SWE-agent isolated ACI from shell-only at +64% relative. Without ablations, you cannot tell which harness primitive did the work.
03
Has the test set been contamination-checked?
SWE-bench Verified shows 35% verbatim 5-gram overlap with training data. Same model on out-of-distribution data may score differently by tens of points.
04
How many seeds and trials?
Claude Sonnet 4.5’s 77.2% is the average of 10 trials. Single-shot figures, especially close ones, sit inside the noise floor.
05
Is the token / compute budget normalised?
Anthropic’s multi-agent lift came with a 15× token tax. Tran & Kiela showed single agents match multi-agent at equal budgets.
06
Has test-set integrity been independently verified?
OpenAI’s own audit found >60% of SWE-bench Verified remaining problems carry critical flaws. The leaderboard preceded the audit.
07
Is the judge methodology disclosed and consistent?
Steel.dev warns that human-eval, WebJudge, and custom agentic judges produce different scores on the same browser agent. 90%+ figures across mixed judges are not comparable.
08
Was the harness tuned on the eval?
A harness optimised against the benchmark it reports against will overstate its lift on tasks that look slightly different.
Synthesis
The thesis, restated honestly
The defensible synthesis the literature supports is narrower than the loudest version of the claim, and more useful.
The harness is the bigger lever for long-horizon, tool-mediated, multi-step tasks where the binding constraint is coordination, memory, or error recovery. SWE-Bench Pro keeps frontier models under 25% on long-horizon coding with a fixed SWE-Agent scaffold; the harness, not the model, is the constraint. [34] GAIA’s 15-to-92 trajectory rides the same dynamic.
The model is the bigger lever for single-shot reasoning and for tasks outside math and symbolic logic, where chain-of-thought, multi-agent debate, and most scaffolding add overhead without lift - or actively degrade performance. Sprague, Zhang, Liu & Kambhampati, and Tran & Kiela form a four-paper consensus on this point.
Both sides agree on one thing: context volume has a negative gradient on performance. Liu et al. 2023 and Chroma 2025 are the receipts. Any harness that grows context unboundedly carries an inherent tax.
Which means the model-vs-harness question is the wrong question. The right one is: for this task class, under these reliability and budget constraints, what is the most leverage we can buy? Sometimes that is a model swap. Sometimes it is a planner. Sometimes it is removing scaffolding the team added by reflex.
“Agentic systems often trade latency and cost for better task performance, and you should consider when this tradeoff makes sense.”
The two quotes that opened this guide were never in tension. They were the same advice from two angles: the harness can be the bigger lever, and it can be the bigger anchor, and the work is telling the two apart on this task, today, with this budget.
References
Sources cited above
Every numeric claim, named study, and quoted line in this guide has a primary-source URL below. Access dates are 2026-05-24 unless noted; figures published behind 403-walled vendor pages were corroborated via the interview / arXiv routes listed.
Yao, S. et al. ReAct: Synergizing Reasoning and Acting in Language Models. arxiv 2210.03629. ICLR 2023 https://arxiv.org/abs/2210.03629
Shinn, N. et al. Reflexion: Language Agents with Verbal Reinforcement Learning. arxiv 2303.11366. HumanEval pass@1 80% → 91% (+11pp) with GPT-4 (paper v4, canonical). An earlier blog post (Reflecting on Reflexion) reported 67% → 88% (+21pp) against OpenAI’s technical-report baseline; we cite the paper pair as it uses the authors’ own re-run baseline and is the peer-reviewed record. NeurIPS 2023 https://arxiv.org/abs/2303.11366
Sprague, Z. et al. To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning. arxiv 2409.12183. ICLR 2025 https://arxiv.org/abs/2409.12183
Liu, R., Kambhampati, S. et al. Mind Your Step (by Step): Chain-of-Thought can Reduce Performance on Tasks where Thinking Makes Humans Worse. arxiv 2410.21333. ICML 2025 https://arxiv.org/abs/2410.21333
Zhang, J. et al. Stop Overvaluing Multi-Agent Debate - We Must Rethink Evaluation and Embrace Model Heterogeneity. arxiv 2502.08788. 2025-02-12 https://arxiv.org/abs/2502.08788
Tran, A. & Kiela, D. Single-Agent LLMs Outperform Multi-Agent Systems on Multi-Hop Reasoning Under Equal Thinking Token Budgets. arxiv 2604.02460. 2026-04 https://arxiv.org/html/2604.02460v1
Yang, H. et al. Auto-GPT for Online Decision Making: Benchmarks and Additional Opinions (AutoGPT is model-conditional). arxiv 2306.02224. 2023-06 https://ar5iv.labs.arxiv.org/html/2306.02224