Building Reliable Agents
Short version: ==A reliable agent is not mainly a better model; it is a better system around the model.== The paper’s whole point is that agents are evaluated like models but deployed like systems, so reliability depe...
Short version: ==A reliable agent is not mainly a better model; it is a better system around the model.== The paper’s whole point is that agents are evaluated like models but deployed like systems, so reliability depends on the harness, execution state, retrieval, memory/state management, permissions, review interfaces, and resource allocation around the model.[:cite[1]{ln=1}][:cite[1]{ln=2}][:cite[1]{ln=3}] It also argues that reliability is a dependency chain: measurement, grading, containment and recovery, retrieval and context, review and accountability, and allocation and cost.[:cite[2]{ln=1}][:cite[2]{ln=2}][:cite[2]{ln=3}][:cite[2]{ln=4}][:cite[2]{ln=5}][:cite[2]{ln=6}][:cite[2]{ln=7}] ==If one earlier layer is weak, later confidence does not fix it.==[:cite[2]{ln=8}][:cite[2]{ln=9}][:cite[2]{ln=10}] What it takes, in practice 1. ==Start with a system level definition of the guarantee.== You need to know what promise the system makes, which component owns it, what state survives failure, and what experiment would falsify the guarantee.[:cite[3]{ln=1}][:cite[3]{ln=2}] The paper explicitly says reliability stops being a model property once runs can outlive workers, wait on services, compete with other runs, mutate shared code, or create durable external effects.[:cite[4]{ln=2}][:cite[4]{ln=3}] 2. ==Build real evaluation before you trust results.== One run is not enough.[:cite[5]{ln=4}][:cite[5]{ln=5}][:cite[5]{ln=6}] You need repeated independent runs, local variance measurement, a predeclared engineering threshold, and enough power to detect the difference that would actually change a decision.[:cite[7]{ln=6}][:cite[7]{ln=7}][:cite[7]{ln=8}][:cite[7]{ln=9}][:cite[6]{ln=5}][:cite[6]{ln=6}] You also need paired comparisons where both systems see the same executable state, permissions, limits, and scoring logic; matching only the task label is not enough.[:cite[8]{ln=1}][:cite[8]{ln=2}][:cite[8]{ln=3}][:cite[8]{ln=4}] 3. ==Make grading execution based, not confidence based.== The document says grades should rest on execution rather than the model’s confidence, and automated graders should be validated against human labels before they gate releases.[:cite[9]{ln=2}] It also warns that agreement with human labels alone does not prove correctness, because agreement can hide shared mistakes or a bad reference target.[:cite[10]{ln=8}][:cite[10]{ln=9}][:cite[10]{ln=10}] In the dependency chain, grading is what turns observations into acceptance decisions, so this gate has to be defensible.[:cite[2]{ln=3}][:cite[11]{ln=1}] 4. ==Contain authority and engineer recovery as first class features.== A reliable run must have bounded access and bounded blast radius.[:cite[12]{ln=1}] The agent’s own account of what happened is only a claim and still needs verification.[:cite[12]{ln=1}] Durable systems record intent and operation identity before external effects, so a replacement worker can retry safely after a crash.[:cite[13]{ln=1}] The boundary must either return the prior result, converge, or force reconciliation.[:cite[13]{ln=1}] The paper also says authoritative facts must live on the work side of the boundary, not only inside the worker.[:cite[14]{ln=1}] And only independently observed evidence should complete logical work, not the worker’s self report.[:cite[15]{ln=1}] 5. ==Test recovery by breaking things on purpose.== The paper is blunt here: recovery should be tested with injected failures, not inferred from architecture diagrams.[:cite[12]{ln=3}] It specifically emphasizes kill point testing across ambiguous commitment gaps and repeated external effects.[:cite[13]{ln=2}][:cite[13]{ln=3}] 6. ==Separate retrieval quality from generation quality.== If the system answers wrong, you need to know whether retrieval failed or reasoning failed.[:cite[16]{ln=1}] The retrieval chain is treated as its own failure path, separating availability, retrieval, placement, contextual sufficiency, and use failures.[:cite[17]{ln=1}] The paper also warns that more agents do not fix a retrieval boundary that treats an empty result as authoritative.[:cite[18]{ln=5}][:cite[18]{ln=6}] 7. ==Treat context as a budget, not a bragging right number.== You should measure how much of the advertised context window is actually usable.[:cite[16]{ln=2}] The document frames retrieval/context as the layer that determines what evidence reaches the agent.[:cite[2]{ln=5}] It also stresses keeping source records rebuildable and derived memory stores versioned with provenance, extraction version, and schema version.[:cite[19]{ln=1}] 8. ==Keep humans where they can actually change outcomes.== Human review should happen where the reviewer can see the risky decision.[:cite[16]{ln=3}] An “advisory” review that cannot block or modify execution is not a real gate; the review decision has to connect to an enforcement point that changes the next reachable...