An agent says it has finished updating a document. The output looks sensible. It even includes a tidy summary of the changes. Then someone opens the document and finds the old version.
There are several possible explanations. The agent might have edited a different copy. A tool might have accepted a request without completing it. A background process might still be running. Or the summary might simply be wrong. From the reader’s perspective, these failures look similar: the promised result is missing.
This is a useful starting point for thinking about agent reliability. Before asking how much work an agent can do, ask what evidence would let another person decide that the work is done.
The example below is deliberately fictional. It uses a small documentation task to explore a general engineering problem. It does not describe a private deployment, a production incident, or a measured improvement.
Start with an observable claim
Suppose the task is to update a public installation guide for a fictional command-line tool. A new release changes one command, and the guide should show the new spelling. The obvious instruction is: update the installation guide.
That instruction leaves several questions unanswered. Which guide? Which release? Does the example need to run? Is changing the source enough, or must the rendered page change too? Who decides whether the update preserves the meaning of the old instructions?
A more useful task description names an observable outcome: the guide for the chosen release contains the new command, the old command no longer appears in the relevant section, and the example succeeds in a clean test environment.
Notice what this does. It turns a broad request into several claims that can be checked separately. A reviewer can disagree with the criteria before any work begins. A tool failure can be reported against a specific claim. The agent has less room to substitute a persuasive explanation for the requested result.
This does not require a long specification. For a small task, a few sentences may be enough. The purpose is to make the meaning of success available outside the agent’s own reasoning.
Keep execution and observation separate
Imagine that a test runner returns a job identifier. That identifier establishes that the request was accepted. It does not establish that the example ran successfully.
The distinction seems obvious when stated directly, yet it is easy to lose inside a long interaction. An agent asks for a test, receives a plausible response, and moves on to writing its final message. The message compresses several events into the phrase “tests passed,” even though the last observed event was submission.
For a background job, the reviewer needs a terminal result and the output that supports it. If the process exits successfully but reports that it skipped the relevant example, that is a different result from exercising the example successfully. If the connection disappears, the result is unknown until it is recovered. Unknown should remain a valid outcome.
The same reasoning applies to file edits and publishing systems. A successful write does not establish that the intended file was changed. A successful build does not establish that the published page serves that build. Each observation supports a particular claim, and it is worth naming that claim explicitly.
Build an evidence table before adding another reviewer
For the fictional guide, a compact review table could contain four rows:
| Claim | Evidence to inspect | What could still be wrong? |
|---|---|---|
| The intended guide changed | A diff of the designated source | The release or section may be wrong |
| The command is executable | Terminal test result and relevant output | The environment may differ from the reader’s |
| The rendered page includes the edit | A fresh render of that source | The deployed page may be older |
| The explanation remains accurate | A reviewer reads the changed section | Important context may still be missing |
These rows are suggestions for this invented task, not a universal checklist. A grammar fix would need less machinery. A change affecting user data would need more careful checks.
The table makes an important gap visible: no single artifact answers every question. A diff is good evidence of a source change. It is weak evidence of runtime behavior. A test result helps with behavior, but says little about whether the prose explains the right concept.
It also keeps verification proportional. If a check has no clear relationship to a claim, reconsider the check. More logs can make a review slower without making its conclusion stronger.
Ask reviewers to look for different failures
A second model can be useful, but agreement alone is a thin signal. If both models read the same incomplete summary, they may both approve the same incomplete work. Changing the model does not automatically change the evidence available to it.
A better review request asks a different question. The author asks how to make the change. The reviewer asks how the change could appear correct while failing the stated goal.
In the guide example, the reviewer might notice that the command appears twice, that the rendered page targets a different release, or that the test inherited a local configuration. These are concrete objections with observable consequences. They are more useful than a generic instruction to double-check everything.
Using different model families is one possible way to diversify review, but it is a hypothesis about error diversity, not proof of independence. Human reviewers can share assumptions too. The practical question is whether the review process discovers failures the author missed, at an acceptable cost.
To investigate that question, keep a small collection of known mistakes. Include a stale render, an edit to the wrong file, a successful process that skipped the example, and an explanation that silently changes the intended behavior. Compare review approaches against those mistakes before making broad claims about reliability.
Treat a synthetic failure as a useful teaching tool
Private logs often contain information that has no place in a public article. Removing a name is rarely sufficient. Timing, paths, unusual wording, and combinations of apparently harmless details can identify a system or reveal how it works.
A synthetic example lets the writer control the boundary. The documentation task in this article preserves a general failure pattern: a completion claim is stronger than the available evidence. It deliberately omits the particulars of any real system.
Synthetic examples have limits. They show that a failure is possible and make a proposed check understandable. They do not establish how often the failure happens, whether a particular system suffers from it, or how much a proposed process improves outcomes.
That boundary should be explicit. If an example is invented, call it invented. If a number is illustrative, label it illustrative. If an improvement has not been measured, describe it as a proposal to test. Readers can learn from an engineering argument without being asked to believe an unsupported success story.
Measure the review process, including its cost
A review process can become expensive without becoming effective. Reviewers may repeat the author’s analysis, generate long comments about minor issues, or block a harmless edit while overlooking a consequential one.
A small evaluation can track several different outcomes: accepted changes that contain a known mistake, rejected changes that are actually correct, time spent reviewing, and the amount of human work required to resolve disagreements. These quantities describe different tradeoffs; collapsing them into a single impressive score hides decisions the team still needs to make.
For an early experiment, use a fixed set of synthetic tasks with documented answers. Record the task, the proposed change, the evidence supplied to the reviewer, and the review decision. Keep a portion of the tasks separate while adjusting the review instructions. Otherwise the process may become very good at recognizing the examples used to design it.
Repeat some trials if the reviewer is stochastic. A single successful review is an observation, not a reliability estimate. Even a carefully built small evaluation will have limited coverage, so write down which failure categories it does not test.
There is no benchmark result hidden in this proposal. The value is in a method that another engineer can inspect and adapt. Its usefulness still has to be established for the work at hand.
A practical first step
Take one recurring agent task and write down its completion claim in ordinary language. Then ask what another person would need to inspect to accept that claim without trusting the final summary.
If the answer is a source diff, collect the diff. If the answer includes execution, collect the terminal result and the relevant output. If the answer depends on meaning or intent, give the reviewer enough context to challenge both. Preserve an explicit unknown state when the evidence cannot be recovered.
You may discover that the task needs another reviewer. You may also discover that it only needs a clearer target or a better observation. Start with that distinction before adding more agents.
The useful question at the end of an agent run is simple: which claims are supported by evidence, and which ones are still open?
Further reading
Anthropic’s Building effective agents describes common workflow patterns and argues for adding complexity only when it helps. This article’s evidence table and fictional documentation scenario are a separate teaching proposal, not a description of Anthropic’s implementation.