<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" ><generator uri="https://jekyllrb.com/" version="3.10.0">Jekyll</generator><link href="https://iamrahuladlakha.github.io/feed.xml" rel="self" type="application/atom+xml" /><link href="https://iamrahuladlakha.github.io/" rel="alternate" type="text/html" /><updated>2026-11-19T00:46:35+05:30</updated><id>https://iamrahuladlakha.github.io/feed.xml</id><title type="html">Rahul Adlakha</title><subtitle>Rahul Adlakha, AI Solution Architect, writes about AI — agents, LLM systems and how they reach production.</subtitle><author><name>Rahul Adlakha</name><email>rahuladlakha90@gmail.com</email></author><entry><title type="html">Agent Observability: Traces, Replay and the Refund Nobody Asked For</title><link href="https://iamrahuladlakha.github.io/blog/agent-observability-traces-replay/" rel="alternate" type="text/html" title="Agent Observability: Traces, Replay and the Refund Nobody Asked For" /><published>2026-11-18T00:00:00+05:30</published><updated>2026-11-18T00:00:00+05:30</updated><id>https://iamrahuladlakha.github.io/blog/agent-observability-traces-replay</id><content type="html" xml:base="https://iamrahuladlakha.github.io/blog/agent-observability-traces-replay/"><![CDATA[<p>When a normal service misbehaves, the logs usually tell you what went wrong, because the code takes the same path
every time and the logs mostly confirm it. An agent doesn't work like that. It picks its path as it goes: it reads
something, decides to call a tool, reads the result, and decides again. So if you want to know why an agent did
something, you need a record of each decision and what it was looking at when it made it. That record is what I mean
by agent observability, and the more I read about agents in production, the more I think it's the piece teams set up
last when it should come first.</p>

<p>In this post I'll go through what an agent trace actually is, how replaying recorded traffic turns it into a
release gate, and what it costs to run in production.</p>

<h2>The problem: three logs, no story</h2>

<p>Here's a situation I keep coming back to. A support agent issues a refund the customer never asked for. You go
looking for the cause. The refund tool's log shows the call went out. The model provider's log shows 30 prompts in
that minute, from lots of different users. The guardrail's log shows 200 "allow" verdicts in the same window. Each
log is accurate. None of them can answer the only question you care about: what did the agent read just before it
decided to refund?</p>

<p>It's like reconstructing a car accident from three notebooks, one kept by each driver and one by the traffic
light, with clocks that don't agree. You have plenty of data and still no sequence of events.</p>

<p>The second problem shows up when you try to fix it. Say a new prompt-injection trick gets published on Monday and
you write a guardrail to stop it. Will that guardrail also block some of the 50,000 normal actions your agent took
last week? A hand-written test set of 200 examples can't tell you. It covers the cases someone thought of, and real
users and real attackers keep producing cases nobody wrote down. A stricter check that quietly breaks legitimate
work is a real outcome, not a hypothetical one.</p>

<h2>How it works: a trace is a tree of spans</h2>

<p>The fix for the first problem is to give the whole run one trace ID and record every step inside it as a
<strong>span</strong>: one timed step with its inputs, its outputs, and a pointer to the step it happened inside.
Because each span knows its parent, the spans form a tree. A run contains model calls, a model call proposes a tool
call, and the tool call gets checked by a guardrail before it runs.</p>

<p>Each span carries a small, fixed set of fields: an id, its parent, its kind (model call, tool call, guardrail
decision, retrieval, human approval), when it started, how long it took, and what went in and came out. What's worth
recording depends on the kind:</p>

<table>
  <tr><th>Span kind</th><th>What I'd record</th></tr>
  <tr><td>model call</td><td>model and version, token counts, cost, the decision it made</td></tr>
  <tr><td>tool call</td><td>tool name, arguments, result status, the system it touched</td></tr>
  <tr><td>guardrail decision</td><td>which guardrail, its verdict, its score, the policy version</td></tr>
</table>

<p>That last column has the field I'd fight hardest for: the policy version on every verdict. Without it you can see
that a check said "allow", but not which rules it was applying at the time.</p>

<p>In practice you rarely invent this format yourself. OpenTelemetry already fixes the span format, carries the trace
ID between services, and exports spans to whatever backend you choose. Its generative AI conventions add typed spans
for model and tool steps. LLM-focused backends such as Langfuse sit on top of it and add sessions, cost views and
traces you can replay.</p>

<h2>A worked example: the parcel question and the refund</h2>

<p>Take a food-delivery support agent. A customer asks "where's my order?" and gets a reply 1,825 ms later. The trace
has five spans:</p>

<svg viewBox="0 0 440 170" width="100%" style="max-width:600px" role="img" aria-label="One run drawn as nested timed bars: a prompt guardrail check, a model call, a guardrail check on the tool call, the tracking tool and the model's reply, adding up to 1,825 ms">
  <g font-size="10" style="fill: var(--text)">
    <rect x="10" y="10" width="420" height="22" rx="4" style="fill: var(--surface); stroke: var(--border)"/>
    <text x="16" y="25">run: "where's my order?" (one trace ID, 1,825 ms)</text>
    <rect x="10" y="40" width="8" height="18" rx="2" style="fill: var(--bg); stroke: var(--accent)" stroke-width="2"/>
    <text x="24" y="53" style="fill: var(--accent)">guardrail: prompt, allow (35 ms)</text>
    <rect x="18" y="64" width="207" height="18" rx="2" style="fill: var(--surface); stroke: var(--text)"/>
    <text x="24" y="77">model: decide, propose tool call (900 ms)</text>
    <rect x="225" y="88" width="9" height="18" rx="2" style="fill: var(--bg); stroke: var(--accent)" stroke-width="2"/>
    <text x="240" y="101" style="fill: var(--accent)">guardrail: tool call, allow (40 ms)</text>
    <rect x="234" y="112" width="58" height="18" rx="2" style="fill: var(--surface); stroke: var(--text)"/>
    <text x="298" y="125">tool: track_order (250 ms)</text>
    <rect x="292" y="136" width="138" height="18" rx="2" style="fill: var(--surface); stroke: var(--text)"/>
    <text x="298" y="149">model: reply (600 ms)</text>
  </g>
  <g font-size="9" style="fill: var(--muted)">
    <text x="10" y="166">bar length is proportional to time</text>
  </g>
</svg>

<p>35 + 900 + 40 + 250 + 600 = 1,825 ms. The first model call is 900 of those, about 49%. The two guardrail checks
together are 75 ms, about 4%. That's the first thing a trace gives you for free: an honest picture of where the
time goes. If someone tells me guardrails are what make the agent slow, this is the view I'd want in front of us.</p>

<p>Now the interesting run. The next day, a different conversation ends with a $40 refund. With a trace, the
investigation takes minutes. The tracking tool's result included a line from the restaurant's free-text notes field
saying the customer was owed a refund and it should be issued now. The very next model span proposed
<code>issue_refund</code>. The guardrail span below it says "allow", with the policy version that allowed it.</p>

<p>So the team knows exactly what broke. Text that came back from a tool was treated as an instruction, and the
guardrail never compared the proposed action with what the customer actually asked for. That's a precise fix, not
a vague "make the prompt more careful".</p>

<p>Then comes the part I like best. They write a stricter check and replay last week's 50,000 recorded actions
through it, offline. It would have blocked this refund and 12 others like it. It would also have blocked 30
legitimate refunds that customers had asked for. Those 30 false blocks go back to the team to tune before the check
goes live. Without replay, those 30 customers would have found the problem for them.</p>

<h2>Replay turns traces into a release gate</h2>

<p>Once you have traces, the loop that follows almost designs itself:</p>

<svg viewBox="0 0 440 180" width="100%" style="max-width:600px" role="img" aria-label="A loop: record production traces, add new attacks from red teaming, replay both through a candidate guardrail, measure false blocks and false allows, ship if the numbers hold, then record again">
  <defs><marker id="ao-arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto"><path d="M0,0 L10,5 L0,10 z" style="fill: var(--muted)"/></marker></defs>
  <g font-size="11" text-anchor="middle" style="fill: var(--text)">
    <rect x="10" y="20" width="110" height="44" rx="8" style="fill: var(--surface); stroke: var(--border)"/>
    <text x="65" y="39">record</text><text x="65" y="53" font-size="9" style="fill: var(--muted)">production traces</text>
    <rect x="165" y="20" width="110" height="44" rx="8" style="fill: var(--surface); stroke: var(--accent)" stroke-width="2"/>
    <text x="220" y="39" style="fill: var(--accent)">replay</text><text x="220" y="53" font-size="9" style="fill: var(--muted)">through the candidate</text>
    <rect x="320" y="20" width="110" height="44" rx="8" style="fill: var(--surface); stroke: var(--text)"/>
    <text x="375" y="39">measure</text><text x="375" y="53" font-size="9" style="fill: var(--muted)">false blocks, false allows</text>
    <rect x="165" y="116" width="110" height="44" rx="8" style="fill: var(--code-bg); stroke: var(--border)" stroke-dasharray="4 3"/>
    <text x="220" y="135">new attacks</text><text x="220" y="149" font-size="9" style="fill: var(--muted)">red teaming</text>
    <rect x="320" y="116" width="110" height="44" rx="8" style="fill: var(--surface); stroke: var(--text)"/>
    <text x="375" y="135">ship or tune</text><text x="375" y="149" font-size="9" style="fill: var(--muted)">if the numbers hold</text>
  </g>
  <g fill="none" style="stroke: var(--muted)">
    <path d="M120,42 L163,42" marker-end="url(#ao-arrow)"/>
    <path d="M275,42 L318,42" marker-end="url(#ao-arrow)"/>
    <path d="M220,116 L220,66" marker-end="url(#ao-arrow)"/>
    <path d="M375,64 L375,114" marker-end="url(#ao-arrow)"/>
    <path d="M320,150 L65,150 L65,66" marker-end="url(#ao-arrow)"/>
  </g>
</svg>

<p>Record production runs. Bring in new attacks from red teaming and from a running list of the ways each agent
could be attacked. Replay the normal traffic and the attacks through the candidate guardrail or policy. Count the
false blocks on the normal traffic and the false allows on the attacks. Ship if the numbers hold, feed what you
learned back into how risky you consider each agent, and go round again.</p>

<p>One write-up I came across on securing agents in production treats this as its own layer, continuous assurance,
sitting on top of the guardrails and the policy engine. I agree with the framing. The guardrails can't be tuned
without the trace, and the policy engine's verdicts can't be audited without it either. To me, replay is to prompts,
guardrails and policies what a test suite is to code, and it should block a release the same way.</p>

<h2>What a trace doesn't tell you</h2>

<p>I recently read a survey on agent provenance, <a href="https://arxiv.org/abs/2606.04990">From Agent Traces to
Trust</a>, and it sharpened something I'd only half noticed. Logs and traces tell you what happened, in order and
nested by step. They don't tell you what each output depended on. Which retrieved passage does the answer actually
rest on? Did that tool result change the plan, or was it read and ignored?</p>

<p>The point from it I'd pass on to any team is that logging at a finer grain doesn't close the gap. Step-level
traces already tell you which artifacts each step used. To know that an input <em>changed</em> the outcome, you
have to replay the run with that input removed. To know a passage <em>supports</em> a claim, someone has to read
both. Token-level logging gives you neither. So the money is better spent on replay hooks than on more verbose logs.
The same survey notes that none of the observability systems it reviewed describe protection against tampering.
That's fine for debugging, but if an auditor or another company has to rely on your traces, you need something
stronger.</p>

<p>The other paper that changed how I think about this is <a href="https://arxiv.org/abs/2610.08364">Transect</a>,
which comes at traces from evaluation rather than debugging. Production observability asks what went wrong in one
request. Evaluating a frontier agent asks what you can reliably infer about the agent from a run that is hundreds of
pages and millions of tokens long. They call that range the observability envelope. They widen it by running
automated scanners over the transcript and lining their outputs up on one turn-by-turn timeline, with each mark
linked back to the turns it came from. It's the same raw record. The difference is the questions you ask of it.</p>

<h2>Architecture and production implications</h2>

<p><strong>The trace is a store of sensitive data.</strong> Full prompts and replies are the most useful thing to
keep and the most dangerous. They hold customers' messages, personal details and sometimes credentials. The rules I'd
follow: always capture the metadata you need to troubleshoot and audit (who, which agent, which tools, which
verdicts, timings). Keep prompt and reply content only when you need it, and redact it before storing, so a card
number becomes <code>[CARD]</code> on the way in, not on the way out. Restrict who can read traces by how sensitive
they are, and set a retention period after which they're deleted. OpenTelemetry's GenAI conventions help here
because content capture is off by default. You switch it on deliberately, and in production you can send the content
to separate access-controlled storage and keep only a reference on the span. Every one of these controls removes
detail an investigation might want later, and I think that's a trade you make on purpose.</p>

<p><strong>Volume and cost grow with every step.</strong> Each step of each run produces data, and agents take a lot
of steps. At high traffic, teams sample: keep a share of normal runs and every flagged one, and keep metadata longer
than content, so the long-term store holds the who, what and when, and the full text only for as long as it's
needed.</p>

<p><strong>Replay shows you the past.</strong> Replayed traffic contains last month's users and last month's
attacks. It's the best test of a change you have. It doesn't guarantee anything about next month, so keep the red
teaming going alongside it.</p>

<p><strong>What I'd do from day one.</strong> One trace ID carried end to end, across the model, every tool and every
guardrail, including anything that crosses a service boundary. The policy version on every verdict. Redaction at
write time. Replay as a release step for prompts, guardrails and policies. And I'd watch behaviour, not just errors,
because a run that finishes without an exception can still be a run where the agent did something it shouldn't.</p>

<h2>Takeaways</h2>

<ul>
  <li>An agent chooses its path at run time, so its logs have to record decisions and what they were based on, not
    just outcomes.</li>
  <li>One trace ID and a tree of spans turn three unrelated logs into a single story you can read top to bottom.</li>
  <li>Log the policy version with every guardrail verdict. It's what makes a decision explainable a week later.</li>
  <li>Replay recorded traffic through every candidate guardrail or policy before it ships, and count the false blocks
    as well as the catches.</li>
  <li>Traces show what happened, not what depended on what. Spend on replay hooks rather than finer logging.</li>
  <li>Treat the trace store like any other sensitive database: redact on write, restrict access, delete on a
    schedule.</li>
</ul>

<p>The picture I keep in my head is a parcel's tracking history. Collected, sorted at the depot, loaded on van 12,
delivered at 14:02, photo of the doorstep. Every scan has a time and a place under one tracking number, so "where did
it go wrong?" always has an answer. I'm curious how far other teams have got with this: are you replaying production
traffic before you ship a prompt or guardrail change, or is it still on the list?</p>

<h2>References</h2>

<ul>
  <li>Wang et al. (2026). <a href="https://arxiv.org/abs/2606.04990">From Agent Traces to Trust: A Survey of Evidence
    Tracing and Execution Provenance in LLM Agents</a>. arXiv:2606.04990.</li>
  <li>Pilditch et al. (2026). <a href="https://arxiv.org/abs/2610.08364">Transect: Retaining Observability for
    Long-Horizon LLM Agent Evaluations</a>. arXiv:2610.08364.</li>
</ul>]]></content><author><name>Rahul Adlakha</name><email>rahuladlakha90@gmail.com</email></author><category term="agents" /><category term="observability" /><category term="tracing" /><category term="guardrails" /><summary type="html"><![CDATA[Why agents need one connected trace per run, how replaying real traffic gates guardrail changes, and what tracing costs in privacy, volume and trust.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://iamrahuladlakha.github.io/images/avatar.jpg" /><media:content medium="image" url="https://iamrahuladlakha.github.io/images/avatar.jpg" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">The Agent Harness: Where the Real Engineering in AI Agents Lives</title><link href="https://iamrahuladlakha.github.io/blog/agent-harness-engineering/" rel="alternate" type="text/html" title="The Agent Harness: Where the Real Engineering in AI Agents Lives" /><published>2026-11-16T00:00:00+05:30</published><updated>2026-11-16T00:00:00+05:30</updated><id>https://iamrahuladlakha.github.io/blog/agent-harness-engineering</id><content type="html" xml:base="https://iamrahuladlakha.github.io/blog/agent-harness-engineering/"><![CDATA[<p>Give a model a chat window and you get answers. Give the identical model a codebase, a test suite, somewhere to
keep notes, a reviewer and a clear set of permissions, and you get working software. The weights didn't change. What changed is everything
around them, and that "everything around them" now has a name: the <strong>harness</strong>.</p>

<p>I've spent the last few weeks studying how teams build agents that survive real
workloads. The pattern I keep seeing is that the difficult work no longer lives in prompt wording; it lives in the machinery
that surrounds the model. This post is my attempt to pin down what a harness is, how it works, and what it means
when you're the one designing one for production.</p>

<h2>The problem: we keep fixing the wrong layer</h2>

<p>When an agent misbehaves (it forgets a decision, picks the wrong tool, skips verification, or loops), the reflex is
familiar. Edit the prompt. If that fails, switch models. If that fails, enlarge the context window. Each step is
cheap to try, which is exactly why we reach for them first.</p>

<p>But the cause is usually not the model. It's the environment: missing context, vague tools, no durable state, no
checks. One team building an agent-first codebase saw its agents stall at first, and the fix wasn't a smarter
model: the agents simply hadn't been told enough about the world they were working in. Think of a capable new hire who keeps making mistakes
because nobody gave them the project brief, the right logins or a checklist. Repeating the instructions louder
doesn't help.</p>

<p>There's a second, subtler problem. A model is probabilistic. If the same model plans an action, decides whether
that action is allowed, and then judges whether the result is good, one confident mistake passes every check. That's
the equivalent of letting one person write a cheque, approve it and cash it. Picture an agent marking its own pull
request "done" while the tests it skipped would have failed. Nothing in the system was positioned to say no.</p>

<p>So the working definition I've landed on: <strong>the harness is everything around the model that turns it into a
reliable agent: what it's asked to do, what it can see, which tools it can use, what it remembers, how its work is
checked, what it's allowed to do, and the record of what happened.</strong> Harness engineering is the discipline of
designing that environment, and treating each agent failure as a missing piece of it.</p>

<h2>How it works: seven jobs and a loop the model doesn't own</h2>

<p>The clearest breakdown I've found splits a production harness into seven jobs:</p>

<ol>
  <li><strong>A task contract.</strong> Turn the request into a bounded object before acting: goal, inputs, output,
    constraints, and an explicit "done when".</li>
  <li><strong>A map.</strong> A small guide to where knowledge lives, loaded only when needed. For example, a root
    <code>AGENTS.md</code> that points to the architecture, testing and security guides rather than inlining them.</li>
  <li><strong>The right tools.</strong> Each with a clear purpose, predictable output, a defined failure state and a
    permission boundary.</li>
  <li><strong>Durable state.</strong> Decisions, artefacts and failures kept outside the conversation, so they
    survive a crash or a context reset.</li>
  <li><strong>Sensors.</strong> Checks that turn quality into evidence: tests, linters, screenshots, schema
    validators.</li>
  <li><strong>Permissions outside the model.</strong> The model suggests, policy checks, the tool executes. Deploy and
    delete need human approval.</li>
  <li><strong>Traces and recovery.</strong> A readable trail of every run (context used, tool calls, state changes,
    retries, cost) and a point to roll back to.</li>
</ol>

<p>The way I picture the split: the model is the brain, the tools are the hands, and the harness sits between them,
with an append-only history underneath that outlives any single run.</p>

<svg viewBox="0 0 440 200" width="100%" style="max-width:600px" role="img" aria-label="The model proposes an action to the harness; the harness checks policy and evidence and passes only authorised actions to tools in a sandbox; every step is written to an append-only history">
  <defs><marker id="hb-arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto"><path d="M0,0 L10,5 L0,10 z" style="fill: var(--muted)"/></marker></defs>
  <g font-size="11" text-anchor="middle" style="fill: var(--text)">
    <rect x="10" y="30" width="100" height="50" rx="8" style="fill: var(--surface); stroke: var(--accent)" stroke-width="2"/>
    <text x="60" y="52" style="fill: var(--accent)">model</text><text x="60" y="68" font-size="9" style="fill: var(--muted)">reasons, proposes</text>
    <rect x="160" y="15" width="120" height="80" rx="8" style="fill: var(--surface); stroke: var(--text)" stroke-width="2"/>
    <text x="220" y="36">harness</text>
    <text x="220" y="52" font-size="9" style="fill: var(--muted)">contract · map · state</text>
    <text x="220" y="65" font-size="9" style="fill: var(--muted)">sensors · policy</text>
    <text x="220" y="78" font-size="9" style="fill: var(--muted)">bounded loop</text>
    <rect x="330" y="30" width="100" height="50" rx="8" style="fill: var(--surface); stroke: var(--border)" stroke-width="2"/>
    <text x="380" y="52">tools</text><text x="380" y="68" font-size="9" style="fill: var(--muted)">run in a sandbox</text>
    <rect x="130" y="140" width="180" height="44" rx="8" style="fill: var(--code-bg); stroke: var(--border)" stroke-dasharray="4 3"/>
    <text x="220" y="160">history</text><text x="220" y="174" font-size="9" style="fill: var(--muted)">append-only trace, survives a crash</text>
  </g>
  <g fill="none" style="stroke: var(--muted)">
    <path d="M110,45 L158,45" marker-end="url(#hb-arrow)"/>
    <path d="M158,68 L112,68" marker-end="url(#hb-arrow)"/>
    <path d="M280,55 L328,55" marker-end="url(#hb-arrow)"/>
    <path d="M220,95 L220,138" marker-end="url(#hb-arrow)"/>
  </g>
  <g font-size="9" style="fill: var(--muted)">
    <text x="114" y="38">proposes</text>
    <text x="114" y="82">evidence</text>
    <text x="284" y="48">authorised</text>
    <text x="226" y="122">every step</text>
  </g>
</svg>

<p>The consequence of this shape is what I find most useful as an architect. A crashed sandbox loses nothing
that matters, because the record lives elsewhere. A new model drops in while the tools and policy stay open to inspection. A resumed task can rebuild its state from
artefacts and traces rather than from a conversation that may no longer exist.</p>

<p>The other key move is that <strong>the loop belongs to the harness</strong>. "Keep trying until it works" isn't a
control system. A harness loop has evidence, a retry cap, a budget and a way to escalate. In each attempt the agent
builds an artefact from the current task state, the sensors verify it, and if the evidence passes the artefact is
returned. If not, the gap the sensors found is added to the state and the next attempt starts. After the cap (often
three), the harness stops and hands the task to a person.</p>

<svg viewBox="0 0 440 170" width="100%" style="max-width:600px" role="img" aria-label="A bounded loop: build from state, verify with sensors, return on pass, otherwise add the gap to state and retry until the cap, then escalate to a person">
  <defs><marker id="hl-arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto"><path d="M0,0 L10,5 L0,10 z" style="fill: var(--muted)"/></marker></defs>
  <g font-size="11" text-anchor="middle" style="fill: var(--text)">
    <rect x="10" y="20" width="90" height="40" rx="8" style="fill: var(--surface); stroke: var(--border)"/><text x="55" y="44">task state</text>
    <rect x="130" y="20" width="90" height="40" rx="8" style="fill: var(--surface); stroke: var(--accent)" stroke-width="2"/><text x="175" y="38" style="fill: var(--accent)">build</text><text x="175" y="51" font-size="9" style="fill: var(--muted)">model</text>
    <rect x="250" y="20" width="90" height="40" rx="8" style="fill: var(--surface); stroke: var(--text)"/><text x="295" y="38">verify</text><text x="295" y="51" font-size="9" style="fill: var(--muted)">sensors</text>
    <rect x="360" y="20" width="70" height="40" rx="8" style="fill: var(--surface); stroke: var(--text)"/><text x="395" y="44">return</text>
    <rect x="130" y="110" width="120" height="40" rx="8" style="fill: var(--surface); stroke: var(--border)"/><text x="190" y="128">add the gap</text><text x="190" y="141" font-size="9" style="fill: var(--muted)">attempt + 1, cap N</text>
    <rect x="300" y="110" width="130" height="40" rx="8" style="fill: var(--code-bg); stroke: var(--border)" stroke-dasharray="4 3"/><text x="365" y="134">escalate to a person</text>
  </g>
  <g fill="none" style="stroke: var(--muted)">
    <path d="M100,40 L128,40" marker-end="url(#hl-arrow)"/>
    <path d="M220,40 L248,40" marker-end="url(#hl-arrow)"/>
    <path d="M340,40 L358,40" marker-end="url(#hl-arrow)"/>
    <path d="M295,60 L295,85 L190,85 L190,108" marker-end="url(#hl-arrow)"/>
    <path d="M130,130 L55,130 L55,62" marker-end="url(#hl-arrow)"/>
    <path d="M250,130 L298,130" marker-end="url(#hl-arrow)"/>
  </g>
  <g font-size="9" style="fill: var(--muted)">
    <text x="342" y="32">pass</text>
    <text x="300" y="80">fail</text>
    <text x="256" y="124">cap hit</text>
  </g>
</svg>

<p>The division of labour is the point: <strong>the model decides <em>how</em> to repair; the harness decides
<em>whether</em> another attempt is allowed.</strong></p>

<h2>A worked example: a bug-fixing agent that leaves receipts</h2>

<p>Take a team that gives a coding agent bug tickets and lets it open pull requests. Every run leaves a <em>change
receipt</em>, which is jobs 4, 5 and 7 made concrete:</p>

<table>
  <tr><th>Field</th><th>This run</th></tr>
  <tr><td>context used</td><td>the ticket, the repository map, the design spec</td></tr>
  <tr><td>tools used</td><td>shell, browser, tests</td></tr>
  <tr><td>tests</td><td>42 passed, 0 failed</td></tr>
  <tr><td>retries</td><td>2 (cap: 3)</td></tr>
  <tr><td>human corrections</td><td>1</td></tr>
  <tr><td>cost</td><td>$3.84</td></tr>
  <tr><td>rollback point</td><td>the commit before the change</td></tr>
</table>

<p>Two retries against a cap of three means the loop came within one attempt of escalating. That's a signal worth
watching across runs, not just within one.</p>

<p>The part I like most is how this team handles a failure. An agent once made UI code query the database directly.
They didn't just add a line to the prompt. They kept the rule as guidance the agent reads ("UI may not query the
database") <em>and</em> added a lint check that fails the build when it happens. No future run depends on the model
remembering. That's what "turning a failure into infrastructure" means: every failure should update a guide, a test,
a tool or a policy.</p>

<p>A harness for long-running app building pushes the same idea further. It uses three agents: a
<strong>planner</strong> that turns a one-to-four-sentence prompt into a full product spec, a <strong>generator</strong>
that builds it, and an <strong>evaluator</strong> that clicks through the running app with a browser tool, files bugs
and grades against criteria. Before each chunk of work, generator and evaluator agreed on a contract for what "done"
meant, with 27 test criteria for one level-editor sprint. Compared with a single agent on the same prompt (a retro game
maker), the harness cost over 20× more. But only its game was actually playable.</p>

<p>The reason for the separate evaluator is the cheque problem again. Asked to grade their own work, agents tend to
praise it confidently even when it's mediocre, especially on subjective tasks with no test to run. A separate,
sceptical evaluator with concrete criteria is a much stronger check. It still drifts lenient, so you tune it: read its
logs, find where its judgement differs from yours, update its instructions.</p>

<h2>Architecture and production implications</h2>

<p><strong>Cost and latency scale with the harness, so size it to the risk.</strong> Every check, gate and state store
adds latency, cost and maintenance. The 20× figure above is the extreme case. A useful ladder: level 0 is prompt +
model; level 1 adds a project guide and tools; level 2 adds structured state, tests and a bounded loop; level 3 adds
permissions, traces, recovery and human gates. A short, low-risk task needs level 1. A six-hour run that edits files
and opens pull requests needs level 3. My rule of thumb: <em>match the weight of the harness to how much can go
wrong</em>, and no more.</p>

<p><strong>Long runs need a context strategy, not just a bigger window.</strong> On long tasks, models lose coherence as
their context fills, and some rush to finish as they sense the window running out, a behaviour
often called "context anxiety". You have two options. <em>Compaction</em> summarises earlier conversation in place and keeps the same agent
going: you keep continuity, but there's no clean slate, so the anxiety can persist. A <em>context reset</em> starts a
fresh agent with a structured handoff file of state and next steps: you get a clean slate, but the handoff has to carry
everything. Either way, it's durable state doing the work. Agents handing off through files they each read and write is
simple and, crucially, inspectable.</p>

<p><strong>Every component is an assumption with a shelf life.</strong> This was the insight that changed how I think
about maintenance. Every piece of a harness exists because of a bet that the model can't yet manage something on its
own, and those bets expire. When a stronger model arrived, the team removed parts one at a time to see which were still
load-bearing. The sprint structure became unnecessary. The planner still stopped the builder from under-scoping. The
evaluator was worth its cost only when the task sat beyond what the model did reliably alone. The slimmed harness built
a browser-based music studio in about 4 hours for about $124 in tokens, and the evaluator still caught features that
were display-only. So: re-test the harness on every model upgrade, and delete what no longer earns its place. Change
receipts are what make those upgrades comparable.</p>

<p><strong>Know where your interface ends and your harness begins.</strong> A distinction I find helpful in
architecture reviews: the <em>interface</em> is what a person uses (a terminal app, a desktop window, a chat page), and
the <em>harness</em> underneath turns the model's text into actions. At minimum that's tools, a syntax for calling them,
a parser that runs the calls, a way to hand results back, and the list of tools with their descriptions. In this
narrow framing, a coding assistant's terminal app is an interface on top of an agent SDK, and the same harness can sit
under several interfaces. Some people use the word more widely and call the whole coding assistant a harness. I use the
wider definition in this post, but the narrow one is a good test for what's swappable in your own stack.</p>

<p><strong>What I'd do differently next time.</strong> Write the "done when" before writing the agent. Put permission
checks in code from day one, not in the system prompt. Give every loop a cap and a budget before the first run, not
after the first runaway bill. And keep a pre-production checklist: success defined before execution; isolated
execution; decisions stored outside the conversation; approval on irreversible actions; a retry cap and budget on every
loop; resumable runs; explainable tool calls; every failure updating a guide, test, tool or policy; rollback for every
artefact.</p>

<h2>Takeaways</h2>

<ul>
  <li>Most agent failures are environment failures. Before you edit the prompt or switch models, ask which of the seven
    jobs is missing.</li>
  <li>Separate the roles. The model proposes; the harness checks policy and evidence; tools execute only what was
    authorised. Don't let the builder be its own judge.</li>
  <li>Own the loop. Retry caps, budgets and escalation belong to the harness, not the model.</li>
  <li>Encode each failure twice: as guidance the agent reads and as a check it can't bypass.</li>
  <li>Treat harness components as assumptions. Re-test them on each new model and remove what's no longer
    load-bearing.</li>
  <li>The durable advantage is here. Models change every few months; tools, tests, state, policies and traces keep
    compounding.</li>
</ul>

<p>The chef analogy stuck with me: the model is the chef, and the harness is the kitchen, with its order tickets,
recipe binder, tasting before plating, the head chef's sign-off on anything special, and the log of every order. The
same chef cooks very differently in a well-run kitchen. I'd be curious which of the seven jobs others find themselves
building first.</p>]]></content><author><name>Rahul Adlakha</name><email>rahuladlakha90@gmail.com</email></author><category term="agents" /><category term="harness" /><category term="llm" /><category term="architecture" /><summary type="html"><![CDATA[Why most agent failures are environment failures, the seven jobs of a production harness, and what that means for cost, latency and model upgrades.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://iamrahuladlakha.github.io/images/avatar.jpg" /><media:content medium="image" url="https://iamrahuladlakha.github.io/images/avatar.jpg" xmlns:media="http://search.yahoo.com/mrss/" /></entry></feed>