All posts

The Agent Harness: Where the Real Engineering in AI Agents Lives

Why most agent failures are environment failures, the seven jobs of a production harness, and what that means for cost, latency and model upgrades.

Give a model a chat window and you get answers. Give the identical model a codebase, a test suite, somewhere to keep notes, a reviewer and a clear set of permissions, and you get working software. The weights didn't change. What changed is everything around them, and that "everything around them" now has a name: the harness.

I've spent the last few weeks studying how teams build agents that survive real workloads. The pattern I keep seeing is that the difficult work no longer lives in prompt wording; it lives in the machinery that surrounds the model. This post is my attempt to pin down what a harness is, how it works, and what it means when you're the one designing one for production.

The problem: we keep fixing the wrong layer

When an agent misbehaves (it forgets a decision, picks the wrong tool, skips verification, or loops), the reflex is familiar. Edit the prompt. If that fails, switch models. If that fails, enlarge the context window. Each step is cheap to try, which is exactly why we reach for them first.

But the cause is usually not the model. It's the environment: missing context, vague tools, no durable state, no checks. One team building an agent-first codebase saw its agents stall at first, and the fix wasn't a smarter model: the agents simply hadn't been told enough about the world they were working in. Think of a capable new hire who keeps making mistakes because nobody gave them the project brief, the right logins or a checklist. Repeating the instructions louder doesn't help.

There's a second, subtler problem. A model is probabilistic. If the same model plans an action, decides whether that action is allowed, and then judges whether the result is good, one confident mistake passes every check. That's the equivalent of letting one person write a cheque, approve it and cash it. Picture an agent marking its own pull request "done" while the tests it skipped would have failed. Nothing in the system was positioned to say no.

So the working definition I've landed on: the harness is everything around the model that turns it into a reliable agent: what it's asked to do, what it can see, which tools it can use, what it remembers, how its work is checked, what it's allowed to do, and the record of what happened. Harness engineering is the discipline of designing that environment, and treating each agent failure as a missing piece of it.

How it works: seven jobs and a loop the model doesn't own

The clearest breakdown I've found splits a production harness into seven jobs:

  1. A task contract. Turn the request into a bounded object before acting: goal, inputs, output, constraints, and an explicit "done when".
  2. A map. A small guide to where knowledge lives, loaded only when needed. For example, a root AGENTS.md that points to the architecture, testing and security guides rather than inlining them.
  3. The right tools. Each with a clear purpose, predictable output, a defined failure state and a permission boundary.
  4. Durable state. Decisions, artefacts and failures kept outside the conversation, so they survive a crash or a context reset.
  5. Sensors. Checks that turn quality into evidence: tests, linters, screenshots, schema validators.
  6. Permissions outside the model. The model suggests, policy checks, the tool executes. Deploy and delete need human approval.
  7. Traces and recovery. A readable trail of every run (context used, tool calls, state changes, retries, cost) and a point to roll back to.

The way I picture the split: the model is the brain, the tools are the hands, and the harness sits between them, with an append-only history underneath that outlives any single run.

modelreasons, proposes harness contract · map · state sensors · policy bounded loop toolsrun in a sandbox historyappend-only trace, survives a crash proposes evidence authorised every step

The consequence of this shape is what I find most useful as an architect. A crashed sandbox loses nothing that matters, because the record lives elsewhere. A new model drops in while the tools and policy stay open to inspection. A resumed task can rebuild its state from artefacts and traces rather than from a conversation that may no longer exist.

The other key move is that the loop belongs to the harness. "Keep trying until it works" isn't a control system. A harness loop has evidence, a retry cap, a budget and a way to escalate. In each attempt the agent builds an artefact from the current task state, the sensors verify it, and if the evidence passes the artefact is returned. If not, the gap the sensors found is added to the state and the next attempt starts. After the cap (often three), the harness stops and hands the task to a person.

task state buildmodel verifysensors return add the gapattempt + 1, cap N escalate to a person pass fail cap hit

The division of labour is the point: the model decides how to repair; the harness decides whether another attempt is allowed.

A worked example: a bug-fixing agent that leaves receipts

Take a team that gives a coding agent bug tickets and lets it open pull requests. Every run leaves a change receipt, which is jobs 4, 5 and 7 made concrete:

FieldThis run
context usedthe ticket, the repository map, the design spec
tools usedshell, browser, tests
tests42 passed, 0 failed
retries2 (cap: 3)
human corrections1
cost$3.84
rollback pointthe commit before the change

Two retries against a cap of three means the loop came within one attempt of escalating. That's a signal worth watching across runs, not just within one.

The part I like most is how this team handles a failure. An agent once made UI code query the database directly. They didn't just add a line to the prompt. They kept the rule as guidance the agent reads ("UI may not query the database") and added a lint check that fails the build when it happens. No future run depends on the model remembering. That's what "turning a failure into infrastructure" means: every failure should update a guide, a test, a tool or a policy.

A harness for long-running app building pushes the same idea further. It uses three agents: a planner that turns a one-to-four-sentence prompt into a full product spec, a generator that builds it, and an evaluator that clicks through the running app with a browser tool, files bugs and grades against criteria. Before each chunk of work, generator and evaluator agreed on a contract for what "done" meant, with 27 test criteria for one level-editor sprint. Compared with a single agent on the same prompt (a retro game maker), the harness cost over 20× more. But only its game was actually playable.

The reason for the separate evaluator is the cheque problem again. Asked to grade their own work, agents tend to praise it confidently even when it's mediocre, especially on subjective tasks with no test to run. A separate, sceptical evaluator with concrete criteria is a much stronger check. It still drifts lenient, so you tune it: read its logs, find where its judgement differs from yours, update its instructions.

Architecture and production implications

Cost and latency scale with the harness, so size it to the risk. Every check, gate and state store adds latency, cost and maintenance. The 20× figure above is the extreme case. A useful ladder: level 0 is prompt + model; level 1 adds a project guide and tools; level 2 adds structured state, tests and a bounded loop; level 3 adds permissions, traces, recovery and human gates. A short, low-risk task needs level 1. A six-hour run that edits files and opens pull requests needs level 3. My rule of thumb: match the weight of the harness to how much can go wrong, and no more.

Long runs need a context strategy, not just a bigger window. On long tasks, models lose coherence as their context fills, and some rush to finish as they sense the window running out, a behaviour often called "context anxiety". You have two options. Compaction summarises earlier conversation in place and keeps the same agent going: you keep continuity, but there's no clean slate, so the anxiety can persist. A context reset starts a fresh agent with a structured handoff file of state and next steps: you get a clean slate, but the handoff has to carry everything. Either way, it's durable state doing the work. Agents handing off through files they each read and write is simple and, crucially, inspectable.

Every component is an assumption with a shelf life. This was the insight that changed how I think about maintenance. Every piece of a harness exists because of a bet that the model can't yet manage something on its own, and those bets expire. When a stronger model arrived, the team removed parts one at a time to see which were still load-bearing. The sprint structure became unnecessary. The planner still stopped the builder from under-scoping. The evaluator was worth its cost only when the task sat beyond what the model did reliably alone. The slimmed harness built a browser-based music studio in about 4 hours for about $124 in tokens, and the evaluator still caught features that were display-only. So: re-test the harness on every model upgrade, and delete what no longer earns its place. Change receipts are what make those upgrades comparable.

Know where your interface ends and your harness begins. A distinction I find helpful in architecture reviews: the interface is what a person uses (a terminal app, a desktop window, a chat page), and the harness underneath turns the model's text into actions. At minimum that's tools, a syntax for calling them, a parser that runs the calls, a way to hand results back, and the list of tools with their descriptions. In this narrow framing, a coding assistant's terminal app is an interface on top of an agent SDK, and the same harness can sit under several interfaces. Some people use the word more widely and call the whole coding assistant a harness. I use the wider definition in this post, but the narrow one is a good test for what's swappable in your own stack.

What I'd do differently next time. Write the "done when" before writing the agent. Put permission checks in code from day one, not in the system prompt. Give every loop a cap and a budget before the first run, not after the first runaway bill. And keep a pre-production checklist: success defined before execution; isolated execution; decisions stored outside the conversation; approval on irreversible actions; a retry cap and budget on every loop; resumable runs; explainable tool calls; every failure updating a guide, test, tool or policy; rollback for every artefact.

Takeaways

  • Most agent failures are environment failures. Before you edit the prompt or switch models, ask which of the seven jobs is missing.
  • Separate the roles. The model proposes; the harness checks policy and evidence; tools execute only what was authorised. Don't let the builder be its own judge.
  • Own the loop. Retry caps, budgets and escalation belong to the harness, not the model.
  • Encode each failure twice: as guidance the agent reads and as a check it can't bypass.
  • Treat harness components as assumptions. Re-test them on each new model and remove what's no longer load-bearing.
  • The durable advantage is here. Models change every few months; tools, tests, state, policies and traces keep compounding.

The chef analogy stuck with me: the model is the chef, and the harness is the kitchen, with its order tickets, recipe binder, tasting before plating, the head chef's sign-off on anything special, and the log of every order. The same chef cooks very differently in a well-run kitchen. I'd be curious which of the seven jobs others find themselves building first.

Comments

Comments load as you scroll here.

Leave a comment

0/2000

No account needed. Comments are reviewed before they appear. No cookies or tracking.