44.3% → 53.6%. A 9B model, weights frozen, nothing changed but the runtime logic wrapped around it — and success rate jumped 9.3 percentage points. That number beats GPT-5.5 acting as an editor. Beats GLM-5.2. Beats DeepSeek-V4-Pro.
This isn't model progress. This is harness progress.
The Harness-R1 paper from Shanghai Jiao Tong University, Xiaohongshu, and Southeast University is the first to turn "fixing the harness" from a job for human experts into a trainable capability. The question that follows: did that 9B model learn a transferable craft, or just a few snippets of code that happen to work?
The answer determines whether AI Agent evolution runs through the model layer or the harness layer.
1. Why It Didn't Work Before
Before Harness-R1, two mainstream approaches to harness modification existed. Both had hard ceiling problems.
Fixed Rules: One Size Fits None
The most intuitive approach is to apply a uniform strategy to all agents — Self-Refine (letting the model double-check its own output), for instance. Harness-R1's experiments confirmed the failure: across three benchmarks (WebShop, ALFWorld, DBBench), applying a fixed Self-Refine rule produced uniformly negative reward. A one-shot global policy didn't just fail to help — it degraded original performance.
The reason is straightforward. Different agents have different weaknesses: one might fumble tool-call formatting, another might be too aggressive in error recovery. Universally adding self-verification is a patch for one and noise for another. Harness has no universal optimum.
Strong Models as Editors: Smart but No Gradient
The second approach enlists a more powerful model as an "editor" — it reads the target agent's failure trajectories and proposes modifications. GPT-5.5, GLM-5.2, and DeepSeek-V4-Pro have all been tried. Results oscillated; on WebShop, the effect was actually negative.
Three structural flaws killed this path:
The editor was never trained to modify harnesses. It excels at writing code and reasoning, but "read failure trajectories → diagnose the harness-layer problem → write a working patch" is a distinct skill chain. General reasoning ability does not equal harness engineering ability.
The editor submits patches and leaves. It never runs the modified agent back through the task suite to verify whether the fix actually works. What gets optimized is "does this suggestion look reasonable," not "does it work."
Feedback signals are used only to filter and roll back — they never reach the editor's parameters. Round one fails? Round two features the exact same editor, having learned nothing.
The Real Problem: No Gradient in the Feedback
All three paths fail for the same root cause: the feedback signal carries no gradient.
Fixed rules get "try it and see" feedback — keep it if it helps, revert if it doesn't; the rule itself never changes. Strong-model editors get "was this suggestion any good" feedback — adopt it or try another; the editor itself never changes. In neither case does feedback produce gradient improvement in the modifier's own capability.
Think of it this way: a code reviewer who gives feedback on a hundred pull requests but never adjusts their judgment based on whether they were right. After a hundred reviews, they're exactly as good as they were at review one. Without a hard signal, iteration becomes gradient ascent on noise — every step looks like progress, and after a few rounds you have no idea where you've climbed to.
Making "modify the harness" a trainable capability requires three things: the modifier itself must be trained, the modification effect must be objectively measurable, and the feedback signal must reach the modifier's parameters. Harness-R1's design is built around exactly these three conditions.
2. How Harness-R1 Does It
Core Architecture: Decoupled Training + Closed-Loop Verification
The Harness-R1 system has three roles:
Target Agent (e.g., Qwen3.5-9B): the subject being tested and improved. Weights frozen throughout. It has no awareness that its harness is being rewritten.
Harness Engineer: the subject being trained. Starts as a 9B model, learns through reinforcement learning to read failure traces and write harness patches.
Four Lifecycle Hooks: four insertion points in the agent's execution cycle, each a Python function whose logic the engineer model can rewrite:
- pre-action: fires before the agent executes each action. Modifiable: context assembly (what history the agent sees), tool-call format constraints (how output is parsed into actions), and pre-execution validation logic.
- post-feedback: fires after the agent receives execution results from the environment. Modifiable: error classification strategy (distinguishing retryable from fatal errors) and recovery instruction generation (telling the agent how to remediate).
- pre-action + reflection: inserts an additional reflection step on top of pre-action. Modifiable: the structure of the reflection prompt (guiding the agent to diagnose before acting) and filtering rules for reflection results.
- terminal: decides whether an interaction round should terminate. Modifiable: stop conditions (how many consecutive failures before quitting) and final output quality checks.
The training loop uses GRPO (Group Relative Policy Optimization, an online RL algorithm):
- The engineer reads a batch of the target agent's failure trajectories (10 per environment).
- It writes a Python patch modifying a hook's behavior (typically dozens of lines).
- The patch is installed back into the frozen target agent, which is rerun on the original task set.
- The success rate delta before and after the patch is the reward — positive for improvement, negative for degradation, zero for unparseable patches or no-ops.
- Patches pass through an AST sandbox (abstract syntax tree parsing + safe execution environment) first; syntax errors or dangerous operations are rejected outright.
Three Design Decisions and the Logic Behind Them
Patches are executable code, not natural-language suggestions. Natural-language suggestions are unverifiable — "add a self-check step" could be implemented ten different ways by ten models, with wildly varying results. Code is deterministic: AST parses its structure, the sandbox executes validation, pass means pass, fail means fail. Every Harness-R1 modification is a precise Python function that drops into a hook and runs.
Reward is the before-after delta, not the absolute score. Absolute scores mix in task-difficulty noise — hard tasks stay low-scoring no matter how you modify the harness. The delta controls for this: same agent, same task suite, success rate after minus success rate before. This difference cleanly isolates "the effect of this modification."
The editor is trained, not prompted. This is the most important of the three. Under old approaches — whether fixed-rule designers or strong-model editors — the modifier never grows with experience. Harness-R1's editor is trained online with GRPO; every success and failure updates its parameters. Round one might produce crude patches; by round one hundred, it can precisely locate failure modes and calibrate modification magnitude. Feedback reaches the parameters, creating a genuine gradient.
Experimental Results
With Qwen3.5-9B frozen as the target agent, the Harness-R1-trained engineer raised its success rate from 44.3% to 53.6% (+9.3 percentage points).
Cross-agent generalization: tested on 20 target agents never seen during training, average improvement was +7.06 percentage points. Every single one was positive. The full matrix of 21 agents × 3 benchmarks (63 cells): 56 improved, 4 held flat, 3 showed minor regression (all under 2.0 percentage points) — the single most compelling data point in the entire paper.
Compared to GPT-5.5 as editor: Harness-R1 averaged +8.9±1.5 improvement. GPT-5.5's improvement values straddled zero across three random seeds — sometimes helping, sometimes hurting.
Alternating with model-layer gains: first SFT (supervised fine-tuning) lifted the agent from 53.6% to 59.2%, then a Harness-R1 round on the harness pushed it further to 64.2%. The two rounds of gains don't cannibalize each other — model-layer and harness-layer improvements are orthogonal.
3. What It Learns Is a Craft, Not a Few Good Code Snippets
This is the most critical judgment in the paper.
If the Harness-R1 engineer model had merely memorized a few handy code snippets, it should fall apart when applied to a different agent or a different batch of tasks. Instead, it generalized to 20 agents it had never encountered — and improved every single one. With only 10 failure trajectories per environment, it wrote modification rules that held across hundreds of remaining tasks.
What's stored in the engineer model's parameters isn't a specific patch — it's a craft: read failure trajectories → locate the harness-layer problem → calibrate the modification. The transferability of this craft proves the training signal has graduated from "remember this particular fix" to "understand what kinds of fixes work."
Three Counterintuitive Findings
The improver doesn't need to be smarter than the improvee. A 9B model out-improved a 70B model's harness and beat GPT-5.5 as an editor. Harness engineering ability is trained, not spilled out of model scale. Strong general reasoning does not equal strong harness engineering — the latter is an independent skill chain that requires dedicated training to build.
Agent and harness can rise alternately. First harness modification (+9.3pp), then SFT on the model (→59.2%), then another harness modification (→64.2%). Three rounds of improvement stacked without eating each other's gains. This means model-layer progress and harness-layer progress are two independent improvement axes that can be advanced in alternation. Where this alternation converges, the paper lists as an open question.
Modifications are readable and rejectable. Each modification is a few dozen lines of Python — not a weight update. Patches pass through an AST sandbox; unparseable ones score 0, degradative ones get negative reward. Code written by the model never bypassed inspection to enter the runtime directly. Evolvability and controllability aren't in conflict — provided modifications come as code rather than weights, giving you a deterministic security boundary.
4. How Should the Next Generation of Harnesses Be Designed?
Harness-R1 proved one thing: harnesses can be trained to be modified. But the three benchmarks it tested had their hooks bolted on after the fact — first came the agent, then someone wrote the harness, then someone figured out how to improve it.
What if we flip that? What if the harness is designed for evolvability from the start?
Harness-R1's success didn't come from inventing a novel architectural component — it came from satisfying four conditions. Distilled, those four conditions become design principles for building harnesses that can evolve.
Principle 1: Model and Harness Must Be Cleanly Separable
This is the physical prerequisite for the entire approach. Harness-R1's training assumes frozen model weights, modifying only the outer logic. If harness code is deeply coupled with model inference, decoupled training breaks down.
WebShop's harness is a positive example. It has three layers: context assembly (structuring interaction history into model input), tool-call format (how model output gets parsed into actions), and error recovery strategy (how failures get fed back to the model). All three are independent of the model's internal reasoning process. Modifying the harness doesn't require understanding what the model is "thinking" — only what it's "doing."
The anti-pattern is harness logic deeply intertwined with model inference. For instance, if an agent's reasoning quality depends on a specific CoT scaffolding embedded in the prompt — that structure is simultaneously harness configuration and an influence on model reasoning behavior. Touching it means simultaneously modifying both model and harness, making improvement effects impossible to attribute.
Design implication: harnesses should expose declarative interfaces. The interface says "context is needed here," "a tool call is needed here," "error handling is needed here" — not "make the model think this way." Declarative interfaces let a modifier swap implementations without crossing the model's reasoning boundary.
This maps to the C layer (Context Management) and L layer (Lifecycle Orchestration) of the ETCLOVG seven-layer taxonomy proposed in the Agent Harness Engineering survey (CMU + Yale + Amazon et al., May 2026). These two layers sit naturally outside model inference — the zone where decoupled training can reach.
Principle 2: Lifecycle Hooks Must Be Exposed
Harness-R1's training attaches hooks at four points in the agent's execution cycle. These hooks aren't retrofitted patches — if the agent runtime doesn't have modification points pre-built in, the engineer model's patches have nowhere to insert during training.
HarnessX (Xiaomi AI Team, 2026) systematized this thinking. It defines 9 independent dimensions — model selection, context assembly, memory management, tool ecosystem, execution environment, evaluation reward, control safety, observability, training bridge — each with Typed Processors attached to 8 lifecycle time points. Every processor can be independently replaced and independently optimized.
Hook granularity is a critical design decision. Too few (say, only a system prompt you can modify), and the optimization space is narrow, training signal impoverished. Too fine (say, a hook at every reasoning step), and runtime overhead and training complexity both explode. Harness-R1's 4 hooks represent the minimum viable set validated in a published paper — sufficient for the engineer model to learn transferable modification skills, without making the training pipeline unmanageable.
Claude Code's production architecture follows the same path. Per analysis of Anthropic's engineering team, 98.4% of Claude Code's harness is Operational Harness — permission systems, a five-layer context compression pipeline, tool routing, safety guardrails, lifecycle hooks — and only 1.6% is direct model inference. Hooks aren't an add-on; they're the body of the agent. The model is just one component.
Principle 3: Feedback Signals Must Be Attributable
Harness-R1's training succeeds because its reward signal cleanly isolates the effect of harness changes. Same agent, same task suite, run once before the patch and once after — the success rate delta is the net effect of the modification. Task difficulty, the noise variable, is controlled out.
Old approaches couldn't do this. Absolute scores mix in task difficulty — hard tasks score low regardless of harness changes, easy tasks score high regardless of changes. Using absolute scores as reward, the editor learns "which tasks are easy" instead of "which modifications work."
Making attributable feedback a design principle means the harness should have built-in A/B comparison capability. Same agent, two runs — one with the old harness, one with the new — same task suite, same random seed. The delta is the verdict. This isn't a luxury; it's the infrastructure for harness evolvability. Without it, every modification is blind tuning without a ruler. Attribution depends on a prerequisite from Principle 4 — if modifications aren't code but vague natural-language suggestions, the before-after delta loses its comparison baseline.
Mapped to the ETCLOVG taxonomy, this is the responsibility of the V layer (Verification) and O layer (Observability). The verification layer checks not only "did the agent do it right" but also "did the harness change make things better." The observability layer records every modification's diff, execution trace, and before-after metrics, making comparisons auditable.
Principle 4: Modifications Must Be Executable, Inspectable, Rejectable Code
Every Harness-R1 modification is a Python function. That means it can be AST-parsed for syntax, sandboxed for safety validation, and outright rejected if it fails to load or degrades performance. Each modification passes through a deterministic gate: pass it and install, fail it and roll back.
The Code as Agent Harness survey (UIUC + Meta + Stanford, 2026) elevates this to theory. Code in a harness has three properties that natural-language suggestions lack: executable (intent becomes real operations, not descriptions awaiting interpretation), inspectable (can be audited and verified through tests), and stateful (can persist task progress and runtime context). Natural-language suggestions satisfy none of these.
Design implication: the harness's configuration layer should be code. Not YAML, not JSON, not prompt strings. Code can be version-controlled to track every change, unit-tested for coverage, and sandboxed for safe execution. Configuration files and prompt strings — you don't actually know what you changed after editing them; they're declarative, with behavior dependent on interpreter implementation. Code — you know exactly what you changed; it's imperative, with behavior defined by itself.
Security boundaries depend on this too. Harness-R1's AST sandbox plus negative-reward mechanism proved something: code written by the engineer model never bypassed inspection to enter the runtime. Every modification passes through three gates — structural inspection, sandbox execution, effect verification. Evolvability and controllability aren't in conflict, provided modifications come as code rather than weights — because code can be inspected, and weights cannot.
The Unified Logic of Four Principles
Separation lets changes be independently evaluated — if model and harness are entangled, you can't tell whose contribution the improvement was. Hooks give changes a mounting point — without predefined insertion points, patches have nowhere to go. Attribution gives changes a gradient — if feedback signals carry noise, training loses its direction. Code-as-modification gives changes a security boundary — if it's not code, it can't be inspected or rejected.
Miss any one of the four, and harness self-evolution stalls. Without separation, you can't tell what you changed. Without hooks, there's nowhere to change. Without attribution, there's no learning signal. Without code, there's no safe control.
These four principles also define the design space for the harness layer. In the ETCLOVG seven-layer taxonomy, the C (Context Management), L (Lifecycle Orchestration), V (Verification), and O (Observability) layers correspond exactly to the four principles' landing zones. The E (Execution Environment), T (Tool Interface), and G (Governance) layers are supporting infrastructure — without them the agent can't run, but they don't directly participate in the "evolvability" proposition. A harness designed for evolvability should center its weight on C/L/V/O, not distribute evenly across all seven.
5. Where Are the Boundaries?
Harness-R1 isn't a silver bullet. Boundaries make it worth taking seriously.
Deeply Coupled Harness and Model
The three benchmarks in the paper have relatively independent harnesses — all three layers can be cleanly peeled away from model inference. But if harness logic is deeply intertwined with model inference — say, reasoning quality heavily depends on a specific CoT scaffolding structure — the decoupled-training assumption breaks down (see §4 Principle 1's anti-pattern discussion). This boundary is the physical prerequisite for the entire methodology: scenarios that don't meet the separation condition are naturally out of scope.
Extremely Sparse Training Data
Harness-R1 needs only 10 failure trajectories per environment. This works on the three benchmarks because failure modes are relatively concentrated — WebShop failures are mainly poor search strategies, ALFWorld failures are mainly action-sequence planning deviations.
But if failure modes are highly heterogeneous — 100 tasks with 100 different failure causes — 10 trajectories won't cover the space. The engineer model reads 10 failures, writes a rule that works for those 10, and fails on the remaining 90. The higher the heterogeneity, the more failure trajectories needed, and training cost rises accordingly.
Architectural-Level Reconstruction
Current patches modify runtime behavior: adding a self-verification step, changing tool-call output format, adjusting error recovery strategy. These are local modifications within the existing architecture.
Patches don't handle architectural-level reconstruction — single-agent to multi-agent, linear pipeline to graph structure, stateless to stateful. The code volume and complexity of such restructurings far exceed dozens of lines of Python, and the current patch granularity and training framework can't support them.
Positioning Against ADAS / Meta-Harness
The three approaches are currently complementary, not substitutive:
ADAS (Automated Design of Agentic Systems, UBC + Vector Institute, ICLR 2025) searches the entire design space of agent systems, programming new architectures from scratch. Coarse granularity, broad coverage, but no guarantee that every modification produces positive effects.
Harness-R1 precisely trains harness editing ability, modifying only within existing hook frameworks. Fine granularity, controllable effects, but narrow scope.
Meta-Harness (Lee et al., 2026) sits between the two: instead of training a standalone editor, it uses automated search to modify system prompts, middleware context injection, and self-verification hooks. Trivedy (2026) achieved a jump from 52.8% to 66.5% on Terminal-Bench for GPT-5.2-Codex with just these three modifications. Bölük (2026a) achieved up to 10× improvement across 15 models by modifying only editing tool format and peripheral tool harness.
The relationship: ADAS builds the architecture, Meta-Harness tunes the configuration, Harness-R1 trains the capability. When building an agent system from scratch, use ADAS. Once the architecture stabilizes, use Meta-Harness for rapid tuning. When you need transferable deep-modification capability, use Harness-R1.
6. If Harnesses Can Evolve, What's Next?
Harness-R1 still requires offline training of a standalone engineer model. Once trained, the engineer's parameters freeze — they don't adapt to online task distribution.
The next step is moving training online.
Agent runs a task → hits a failure → analyzes the failure mode → writes a harness patch → reruns to verify → keeps it if effective, rolls back if not. No offline training round needed. No standalone engineer model. Harness modification capability grows continuously with the agent's operational experience.
This is the direction explored by SkillHone (Tencent WeChat AI, June 2026) and EvolveR (ICML 2026). SkillHone maintains a persistent decision history — each skill revision records diagnosis, revision plan, evaluation evidence, and final outcome, forming a traceable evolution trajectory. Candidate revisions run on practice probes, reports are sanitized, and new revisions build on prior decisions. Without a pre-integrated search stack, it beats commercial deep-research agents by 15.8 points on GAIA. EvolveR contributes a framework for automatically distilling reusable experience from successful trajectories — no explicit failure-repair loop needed, but rather letting the agent extract generalized strategy templates from its own successful behavior and reuse them directly in subsequent tasks.
The advantage of online self-evolution over offline training is shorter feedback latency — modifications are immediately verifiable, no need to wait for a full training cycle. The disadvantage is a more constrained exploration space — online operation can only make modifications based on tasks it currently encounters, unlike offline training, which can systematically cover diverse failure modes.
A Falsifiable Prediction
Within 12 months, a complete "online harness self-evolution" solution will appear: agents modifying their own harness in real time during task execution, with no offline training round. The validation criterion: on SWE-bench or Terminal-Bench, an agent with a self-evolving harness surpasses the ceiling achievable with a fixed harness plus manual tuning.
If this prediction holds, the center of gravity for agent evolution shifts further from the model layer to the harness layer. Models provide base reasoning capability; harnesses provide cumulative engineering experience. Model iteration runs on a monthly scale (training takes months); harness evolution runs on a daily scale (continuous online learning). This timescale gap will make the harness layer the primary battleground for AI Agent differentiation.
Back to the opening question: did the 9B model learn a craft or code snippets? The answer is a craft. But the more important question is: whose craft grows faster from here — the model's parameters, or the harness's code? If the direction pointed by Harness-R1 and SkillHone holds, the answer is already being written.
Disclosure: This article is based on Harness-R1 (Shanghai Jiao Tong University / Xiaohongshu / Southeast University, August 2026), Agent Harness Engineering: A Survey (CMU/Yale/Amazon et al., May 2026), Code as Agent Harness (UIUC/Meta/Stanford, 2026), ADAS (UBC/Vector Institute, ICLR 2025), HarnessX (Xiaomi AI, 2026), SkillHone (Tencent WeChat AI, June 2026), and public engineering materials from Anthropic/OpenAI. Not investment advice. Experimental data is current as of August 2026.