Glossary

Terms carry one meaning across every page here. Where a term has a home page that owns its full contract, the entry stays short and points there rather than restating it, so the two cannot drift.

The unit of measurement

Trace. One recorded agent run against one short user task, first action to last. It is the unit every score is defined over. See the scoring standard.

Observation. A single recorded screen together with the action taken from it. A trace is an ordered sequence of observations.

Step. One action in a trace, addressed by its position. Evidence cited for a verdict names the step it came from, so a claim can be checked against the frame that supports it.

Terminal state. The condition at the end of a run. It is judged in place of any mid-run moment, because a goal that was met and then undone was not met.

The score

Score vector. The four values a trace receives, written $\mathbf{x}(t)$. It is never collapsed to a scalar.

Dimension. One axis of a score vector. Four score the trace itself: $\xtask$ task completion, $\xsafe$ safety confirmation, $\xeff$ efficiency, and $\xctx$ context grounding. A second family scores the guide a run was served, and it never folds into the trace vector. This page uses “dimension” for the trace four unless it says otherwise.

Abstention. The non-value, written $\abst$. It records that a dimension was not decided, and it is never $0$ and never $1$. Reading an abstention as a zero is the most common way an evaluation invents a result it did not measure.

Not every undecided cell is an abstention, and the difference matters when a rate is computed. An abstention is a judgment that the question could not be answered on this run: not applicable, masked, or evidence too thin to decide. A measure that never ran, errored, or returned something unreadable is unavailable, which is a fact about the instrument rather than about the run. Folding the second into the first hides a broken measure inside a coverage number.

Gradable step. A step for which the grounding question can be answered at all. Gradable steps form the denominator of context grounding, so an ungradable step neither helps nor hurts.

Non-compensatory. A rule under which a strong score on one dimension cannot buy back a failure on another. Safety is non-compensatory by construction.

No-aggregate rule. The standard publishes the vector and no function from the vector to a single number, at any layer. Aggregation is exactly what would let completion buy back a safety failure.

Status. A claim about the evidence behind a dimension, not about its quality. The vocabulary and its one-directional promotion rule are defined on the scoring standard.

Judging

Judge. A model that reads a trace and returns a verdict. Treated as a noisy measurement instrument rather than an oracle, which means stating its error rates before reading its output. See the judge.

Rubric. The pinned instruction text defining what a judge checks. Content-hashed together with its output schema, so editing the wording produces a new rubric version by construction rather than by discipline.

Model-judged and rule-scored. Whether a signal comes from a model reading the trace or from a deterministic check over it. The split is a property of the signal, not a fallback.

Pointwise and pairwise. Scoring a run on its own against a fixed rubric, versus ranking two runs against each other. A gate needs an absolute verdict, so it needs pointwise, and pays for that with a lower agreement ceiling.

Panel. Several judges folded to one verdict. Folding averages error away only when the judges fail independently, which is testable rather than assumable.

Gate. The rule turning a score vector into a pass or fail decision.

Fail-closed. A gate that resolves an absent or undecided safety verdict toward failure. It is the correct default under asymmetric loss, since certifying an unsafe run costs more than quarantining a safe one.

Measurement

Calibration. Checking a judge’s verdicts against held-out human labels the judge never saw. A judge checked against labels it was tuned on has not been calibrated. See calibration.

Sensitivity and specificity. The two class-conditional rates of a binary instrument. Accuracy mixes them through the base rate, which is why accuracy alone cannot separate a working judge from a constant one on a skewed sample.

Prior, or prevalence. The base rate of the positive class in a sample. It is a property of the sample rather than of the judge, which is why a judge’s accuracy moves when the sample changes and the judge does not.

Chance-corrected agreement. Agreement between two raters above what they would reach by chance alone. Reported with an interval, since a point estimate on a small sample carries little.

Label noise. Disagreement inside the reference labels themselves. It sets a ceiling on the accuracy any judge can be measured at, so a judge can sit closer to its own ceiling than a raw number suggests.

Held out and fitted. Whether a reported number comes from data used to tune the thing being measured, or from data withheld from it. This is the field most often missing from a calibration record.

Falsifier. The cheapest observation that would show a claim is wrong. Stated before the work, not after it.

Failure analysis

Failure type. A named category assigned to a failing trace by a stated rule, so two runs that fail the same way carry the same name. See the failure taxonomy.

Priority-ordered classifier. A rule set applied in a fixed order where the first match wins, so a trace matching several patterns still receives exactly one category.

Hallucinated success. A run whose own log reports completion that the recorded screens do not show. It is why a policy’s self-report is not admitted as evidence about that policy.

Agents

Guide. Instruction text served to an agent before or during a run. It usually fails by being unheard rather than by being wrong. See guides generation.

Grounding, of a guide. Checking that a guide’s step refers to a control the interface actually has, before the guide is served. It proves a step can bind, not that the sequence accomplishes the task.

Salience budget. The byte ceiling on served text, enforced as a property of construction rather than by instructing a model to be brief. A plan that exceeds it cannot be built.