Architecture

An agent that cannot be measured cannot be improved, and whether it can be measured is decided by its architecture rather than by the evaluation built on top. This page is about that seam: what the stack has to emit, and why each requirement exists.

The loop

Observe. Capture the environment state the policy will act on, and capture it as the policy saw it rather than as it was later reconstructed. A record assembled after the fact cannot distinguish a policy that acted on bad information from one that acted badly on good information.
Decide. Select the next action from that observation and whatever the policy carries across steps. This is the only step where the model is free; the others are contracts.
Act. Apply the action to the environment, and record what was targeted separately from what was hit. The gap between those two is where a large share of failures live, and a record that collapses them makes those failures invisible.
Emit. Append the observation, the action, and the resulting state to the record, with the step’s position. The position is what lets evidence be cited later.

Nothing in the loop scores anything. Scoring is a separate layer reading the record, which is what allows the same record to be scored again under a revised standard.

What makes a record gradable

A record is gradable when a reader who was not present can reach the same verdict as one who was. That imposes four requirements, and each of them fails in a specific way when it is missing.

Requirement	Missing it means
The state is observable at each step	A verdict has to be inferred, so two graders reach different answers on identical runs
Intent and outcome are recorded separately	A wrong target and a failed action look identical
Steps are addressable by position	Evidence cannot be cited, so a verdict cannot be checked
The terminal state is captured, not just the last action	A goal met and then undone reads as met

The last row is the one most often missed, and it is the one that inflates results. An agent that succeeds and then keeps acting has a terminal state that no longer shows success, which is the correct verdict and the one a last-action record cannot produce.

The self-report boundary

The policy’s own account of what it did is not evidence about what it did. It is an output of the same system under evaluation, so admitting it as proof lets the system grade itself.

This has a concrete architectural consequence: the record has to carry observed state, not only the policy’s narration of state. A stack that records “the agent reported completing the task” has recorded a claim. A stack that records the screen after the final action has recorded evidence. Only the second supports a verdict, and the difference is invisible until someone checks.

The same boundary applies in reverse. A trajectory that reads as confused is not evidence of failure if the terminal state is correct, and the record needs to support that reading too. See the judge for how the two are kept apart at scoring time.

Where scoring attaches

The scoring contract is defined over a completed record, so it can run after the fact on a stored corpus or as a run completes. Nothing in the contract should depend on which: a metric that only works offline cannot gate anything, and one that only works online cannot be recomputed when the standard changes.

Two properties follow when that holds. A stored record can be rescored under a revised standard without rerunning the agent, which is what makes a standard revisable at all. And an online score is comparable to an offline one, because it is the same function of the same record.

This is a requirement, and it is the one most easily lost. Path independence does not survive a second dispatcher. Where the online and offline routes are separate code paths that happen to terminate in the same scorers, they can diverge in what they pass those scorers, and a dimension whose inputs differ between paths is silently no longer the same measurement. The failure is quiet because both paths return a well-formed score. Anything that reaches a scorer as optional context rather than as part of the record is where this starts, so the check is not “do both paths call the same scorer” but “do both paths hand it the same arguments”.

State that crosses steps

Whatever the policy carries between steps is part of the input to every later decision, so a record that omits it cannot explain the decisions. It does not have to be interpretable, and often is not. It does have to be present, or the record supports description and not diagnosis.

This is the difference between knowing that a run failed at step nine and knowing why step nine was reachable at all.

Named gaps

The task distribution is outside this boundary. The architecture governs how a run is recorded, not which tasks exist. A biased task set produces perfectly gradable records of the wrong thing, and nothing here can detect that. See the scoring standard, which names the same gap from the other side.

Observability is not free and its cost is not measured here. Capturing state at every step changes the run’s timing, and a policy whose behavior depends on timing is therefore observed under conditions slightly unlike its deployed ones. The size of that effect is unmeasured, which makes it a known unknown rather than a bounded one.

A gradable record is not a fair one. Every requirement above concerns whether a verdict can be reached and checked. None of them makes the verdict correct; that is what calibration is for.