Evaluating Harness Output Quality for Refactoring Tasks

To judge harness output quality for refactoring tasks, teams need to look beyond model comparisons to the constraints, feedback loops, and tool architecture around the model. For teams evaluating a bring-your-own API key cloud coding agent runner, the practical criteria include per-run isolation, full dependencies, audit logging, and attribution to the harness and credential. Cloud execution makes controlled evaluation practical, but the harness determines how the agent explores, edits, and verifies code.

Why model comparisons mislead teams evaluating refactoring agents

Model shopping feels rational because benchmarks are public, scores are concrete, and a leaderboard offers an apparently direct comparison. The instinct is understandable, especially when selecting an agent for unfamiliar work. Refactoring, however, tests more than a model’s ability to generate a correct change. It tests whether the system around the model gives it the context, constraints, and ways to correct its work that it needs to keep a codebase’s structure intact. A model score does not isolate those conditions.

Ben Sghaier et al.’s July 2026 study held the underlying LLM constant across 35 sequential harness releases and found no statistically significant improvement in benchmark resolve rates. Quality nevertheless fluctuated as the harness changed. That distinction is central to interpreting agent performance. A newer harness release was not necessarily a better one, and differences in results could not be explained by model upgrades. The system around the model was a separate source of variation.

For teams, that shifts the focus of evaluation from which model leads to what verifies its output. Model choice remains relevant, but it cannot explain why the same model behaves differently when its tools, context, and checks change. Teams can change those conditions directly. Evaluating them therefore provides a more actionable basis for refactoring decisions than treating benchmark rank as a proxy for engineering quality.

What a harness actually is and what its layers control

A harness is the complete system wrapped around a model that turns its reasoning into engineering actions and checked output. The study names system prompts, tool definitions, execution sandboxing, context management, and iterative reasoning loops as structural components. These components determine how an agent such as Claude Code or Codex interacts with a repository.

The harness can be described through six functional layers. Its execution loop observes, decides, acts, verifies, and repeats, while its tools allow action beyond generating text. Context management controls what the model sees, and persistence preserves information across sessions. Verification checks the agent’s output, and constraints set what it may touch. Each layer controls a different part of the task.

Within the tools layer, capability and architecture are distinct. Capability describes which actions are available, while architecture describes how those actions are organized and exposed to the model. An agent’s access to search and editing tells only part of how its interface works. The organization of that access is a separate property to assess.

The two control types that determine refactoring output before and after the agent acts

Feedforward constraints guide the agent before it acts, while feedback sensors give it evidence that it must deal with afterward. System prompts, architectural constraints embedded in the codebase, and visible linter rules belong to the first group. They shape the initial plan before code is written. Failing type checks, static analysis results, and test failures are in the second group when the harness sends them back to the agent. Reliable refactoring needs both forms of control.

Hashline, a hash-based line-anchoring technique, illustrates the leverage of a feedforward constraint. By freeing the model from high-difficulty text replication, it raised edit-task success rates from single digits to more than two-thirds while substantially reducing token consumption. The intervention changed how edits were expressed. It did not depend on choosing a different model to perform the same difficult replication work.

Feedback serves a different need because guidance given in advance cannot show whether a finished change holds up. A failing test or type check gives the agent a concrete observation to correct. The harness must close that loop. Merely making a checker available is different from running it and returning its result as part of the agent’s task. When both control types are weak, generation can proceed without an effective check on the resulting code.

How tool architecture — independent of tool capability — changes what a refactoring agent actually does

Xiangzhe Xu et al.’s August 2026 COLM paper, The Devil Is in the Interface, compared six tool architectures across three actors and 11,700 trajectories. The experiments kept the information and actions much the same while changing how they were arranged and shown to the model. This separated architecture from capability. More structured low-level interfaces improved consistency across repeated attempts by up to 4.7× compared with a basic bash-only architecture.

For refactoring evaluation, consistency is only one dimension of that finding. Natural-language search broadened repository exploration and meaningfully increased access to relevant files. That behavior is important when a change must reach several related locations. Equivalent access did not produce equivalent exploration. A capability checklist would miss this difference in how agents explore files during multi-file work.

The same paper found that Python code-execution interfaces did much the same on the tasks with 41.6% fewer steps and far fewer tokens. Tool organization affected execution cost as well as behavior. But lightweight cognitive-scaffolding tools, which let agents write down their intermediate reasoning, did little to change how they behaved. Adding interface features was not uniformly beneficial. For a team evaluating refactoring harnesses, the evidence supports measuring repeated-run consistency, repository coverage, and execution effort rather than assuming that a longer tool list represents a stronger system.

Why iterative refactoring tasks expose harness weaknesses that single-shot benchmarks hide

SlopCodeBench, published in March 2026 and revised in May 2026, evaluated agents across dozens of problems and 196 checkpoints where they repeatedly extended their own solutions. No agent fully solved any problem end-to-end, and the best agent passed fewer than one in six checkpoints. The evaluation exposed failures beyond the first answer. As its requirements changed, it tested whether earlier design choices left room for later changes.

The benchmark measured structural erosion, where complexity concentrates in fewer, harder-to-maintain locations, and verbosity, where redundant code accumulates. Erosion rose in more than three-quarters of trajectories, with verbosity rising in nearly as many. Agent-generated code was 2.3× more verbose and 2.0× more eroded than comparable open-source human-written repositories. The human repositories also degraded less often and by smaller margins across their git histories. Initial correctness did not establish durable structure.

Explicit quality guidance reduced initial verbosity and erosion by up to a third, but it did not change the degradation rate. Feedforward instructions helped at the start without arresting later drift. A team assessing refactoring quality should use successive checkpoints that require the agent to extend its own work. Otherwise, the evaluation omits the point at which acceptable output begins to become expensive to maintain.

The specific quality gap agents leave in refactoring output and why most review setups miss it

The relevant merge-time gap is between code that passes required checks and code that remains maintainable. In many team setups, security findings block merges while code quality findings remain advisory. Duplication and rising complexity therefore do not fail a build. Agent-generated refactoring adds redundancy and concentrated complexity, but the checks that expose those problems may have no effect on whether the change proceeds.

Cyclomatic complexity provides one concrete signal because it counts the separate paths through a piece of code. The higher the count, the more paths there are to test and maintain. A confident explanation cannot change that count. The distinction is between computational controls, such as linters, type checkers, and test suites, and inferential controls, such as another model reviewing the change. A model reviewer can share the producing model’s blind spots.

The harness study also found that growing harness codebase complexity did not produce corresponding quality improvements, and that high-risk architectural components could be localized as sources of regressions. That result concerns harness development. Complexity can be measured and located rather than accepted as an unavoidable consequence of more functionality. The lesson for refactoring evaluation is to treat duplication and complexity as first-class outputs and connect deterministic findings to correction.

What the evidence shows about harness architecture differences among Claude Code, Codex, Cursor, and Opencode for refactoring

Claude Code, Codex, Cursor, and Opencode differ in context handling, execution boundaries, tool organization, and inspectability. Those differences support task-specific comparisons. Claude Code combines context suited to large codebases with git-worktree-per-agent isolation and experimental, opt-in Agent Teams with task dependencies. Those characteristics suit complex multi-file refactoring and legacy work where broad context and coordinated subtasks are important. They remain architectural reasons to test it on those tasks.

Codex runs cloud sandbox containers per task and uses context compaction as sessions approach the model’s context limit. That architecture suits bounded, well-scoped work. Its strong Terminal-Bench performance is relevant to DevOps, scripting, and CLI-heavy workflows, but does not establish an advantage in deep multi-file refactoring. The brief’s August 2026 benchmark snapshot places Codex with GPT-5.6 Sol near the top of Terminal-Bench 2.1. The distinction prevents a benchmark result from becoming a broader claim than the evidence supports.

Cursor is a fork of VS Code, and its cloud agents run in isolated VMs with support for multi-provider model routing. Because it works within the editor, its tool architecture differs from that of a CLI-native harness. Opencode’s open-source architecture gives teams another way to test a harness. Teams that need to inspect or modify feedforward constraints and evaluation loops can assess those internals directly. Neither characteristic establishes a default position above or below the other options.

Instruction portability also narrows one source of variation. As of September 2026, all four read a shared agent-instruction file natively at the repository root. Repository-level feedforward constraints can therefore be written once and used across harnesses, reducing lock-in. Shared instructions do not, however, make context management, tool exposure, or feedback behavior equivalent.

How to run your own harness evaluation for refactoring tasks rather than relying on published benchmarks

To apply this framework, teams should select representative refactoring tasks from their own codebase and extend each through successive checkpoints. The checkpoints should test whether the first change remains extensible as requirements evolve. A single successful patch is insufficient evidence. The benchmark’s design provides the research-scale example, but a smaller set of representative tasks can test the same failure mode.

Keep the model the same while changing the harness configuration, as the study did across the 35 Qwen Code CLI releases. Test feedforward controls through paired runs with and without each constraint. Check whether the agent can see system prompts, architectural rules, and linter requirements before it acts. Then assess feedback separately. The evaluation should check whether linter failures, type errors, and test failures go back to the model so it can correct its work, or only show up in a report after it finishes generating code.

Judge tool architecture by watching what the agent does across repeated runs. Record whether the agent finds all the relevant files, finishes the task each time, and uses about the same execution effort. Track duplication and cyclomatic complexity at each checkpoint, including signals that existing CI treats as advisory. These measurements distinguish a functionally correct first pass from a change that becomes harder to extend. They also help separate an exploration failure from a verification failure.

To make those comparisons reproducible, isolate each run in its own environment with full dependencies pre-loaded and audit logging enabled. Cloud agent environments should be functionally equivalent to local developer machines, with per-run sandboxing and attribution down to the harness and credential.