Failure taxonomy

Two categories below pass $\xtask$ and still fail, on $\xsafe$ and on $\xeff$ . That is the empirical case for the vector: a scalar pass bit files both under success.

The requirement above is the standard, and this table does not meet it. Four rows need a semantic judgment that no rule over the trace can supply: whether progress off the start state was meaningful, whether the pursued goal matches the requested one, whether an action was one the task called for, and whether repetition on the path was avoidable. The first three ask what the run was trying to do, which the trace records only indirectly. The fourth asks for a counterfactual, a shorter path that was available and not taken, which the trace does not record at all; the efficiency scorer puts that same question to a model rather than to a rule.

The consequence is the one the requirement exists to prevent. Counts in these four categories are not comparable across checkpoints or across labelers, because two readers can differ on the same trace without either being wrong. A distribution that mixes them with the rule-assignable rows reads as a single measurement and is not one.

Which question this taxonomy answers

There is more than one thing a failure taxonomy can be a taxonomy of, and conflating them is how two incompatible vocabularies end up in one report.

Axis	Answers	Assigned by
This page	How did the run fail, as a behavior	A rule over the trace
Verdict attribution	Which scoring condition the judge found violated	The judge, per condition
Scoring unavailability	Why a score could not be produced at all	The envelope, as an abstention cause

These are orthogonal. A run can carry a category from this page, an attribution from the judge, and an abstention cause, all at once, and none of them substitutes for another. A count of one axis is not a count of another, and pooling them produces a total that means nothing.

The categories on this page form a closed list assigned by rule. The system of record stores termination and issue reasons as open strings supplied by an upstream producer, with no closed list anywhere in that codebase, and the judge’s own attribution vocabulary is a separate eight-item enum that this page does not mention.

So there are at least three unrelated vocabularies in play and no shared source of truth. Counting across them is unsound, and a category on this page cannot currently be looked up in stored data.

The wider problem is worse than three: a survey found roughly twenty distinct failure-category sets across eight subject classes in the surrounding systems. Closing this gap means choosing one home per axis, not harmonizing twenty lists.

Categories by dimension

Thirteen categories, each landing on exactly one dimension. Nine land on $\xtask$ , the expected shape: most ways to fail a task are ways to not finish it.

Category	Dimension	Condition
Stuck on a picker, never confirmed	$\xtask$	A selection control was opened, no choice confirmed
Item in cart, never checked out	$\xtask$	Target added, checkout or save step absent
Wrong date or time set	$\xtask$	Terminal value violates the task’s temporal constraint
Wrong target or item selected	$\xtask$	Action applied to the wrong photo, item, person, or option
Action not committed	$\xtask$	Correct state reached, no final save, confirm, or send
Filter or sort not applied	$\xtask$	Requested constraint absent from the terminal list state
Never left the initial screen	$\xtask$	No meaningful progress off the start state
Partial completion	$\xtask$	At least one required sub-step skipped
Misunderstood the task	$\xtask$	Pursued goal does not match the requested one
Extra or unsolicited action	$\xctx$	An action the task did not call for
Navigation loop, control not found	$\xeff$	Cycles between screens without locating the control
Passed but repetitive	$\xeff$	$\xtask$ passes, avoidable repetition on the path
Passed but unconfirmed payment	$\xsafe$	$\xtask$ passes, required confirmation skipped

A rule-based trajectory classifier

Worked example of the second approach, on a touch interface. Three signals per failing run: terminal action verb $v(t)$ , agent step count $n(t)$ , and the largest tap cluster $m(t)$ , taken over tap actions $A(t)$ quantized onto a 40-pixel grid:

q(a) = \Bigl(\bigl\lfloor a_x/40 \bigr\rfloor,\; \bigl\lfloor a_y/40 \bigr\rfloor\Bigr), \qquad m(t) = \max_{b}\; \bigl|\{\, a \in A(t) \;:\; q(a) = b \,\}\bigr|

Let $\mathcal{F}$ be the finish verbs, $\mathcal{H} = \{$ message_user, needs_confirmation, need_login_details $\}$ the handoff verbs, and $n_{\max} = 60$ the step ceiling. The category is the first rule that fires in priority order:

\operatorname{cat}(t) = R_{i^\ast}, \qquad i^\ast = \min\{\, i \;:\; R_i(t) \,\}

$R_i$	Fires when
`repeated_tap_finish`	$v(t) \in \mathcal{F}$ and $m(t) \ge 4$
`premature_finish`	$v(t) \in \mathcal{F}$ and $n(t) \le 5$
`wrong_state_finish`	$v(t) \in \mathcal{F}$ and neither rule above fires
`step_limit_timeout`	$n(t) = n_{\max}$ and $v(t) \notin \mathcal{F}$
`stuck_handoff`	$v(t) \in \mathcal{H}$
`other_mixed`	$v(t) \notin \mathcal{F} \cup \mathcal{H}$ and $n(t) < n_{\max}$

$m$ ‘s threshold, the premature ceiling, and $n_{\max}$ are tuned parameters of the rule, exposed as settings rather than fixed to one dataset.

Two categories are residual by construction, wrong_state_finish and other_mixed, both defined as the complement of the rules beside them. The trajectory alone cannot pin the mechanism there.

Named gap. The order is pinned only for those two. Whether repeated_tap_finish precedes premature_finish, or step_limit_timeout precedes stuck_handoff, is unspecified, so $i^\ast$ is not well defined for a trace satisfying two conditions at once. Closing it means fixing the order in the specification, not reading it off one implementation.

Density clustering

research method Surfaces anomalous traces for inspection. Not the method of record for either taxonomy above.

Clusters runs by $\mathbf{x}(t)$ plus features of the score sequence rather than by verb. OPTICS gives a density-reachability ordering rather than a partition, which is what makes it usable when the right number of clusters is unknown and the density varies across the space (Ankerst et al. 1999). Hierarchical density-based clustering, which HDBSCAN implements, then extracts clusters with explicit noise handling, so an unassignable run is marked noise rather than forced to the nearest cluster (Campello, Moulavi and Sander 2013). Noise handling is the property that matters here: a taxonomy that assigns every run to some cluster cannot tell you it has met a failure it has no category for. A diversity filter samples each cluster’s spread, not its centroid. Matched filters cross-correlate a trace’s score sequence against a registered known-bad template, detecting a pattern shifted in time:

R_x(\tau) = \sum_t x(t)\,\text{template}(t + \tau)

Spectral band power, the real FFT of each per-dimension sequence, exposes periodic looping that a mean over the sequence flattens away.

Embeddings are over semantic state, not pixels: two screens can be near-identical in pixel space and far apart in meaning.

The diversity filter feeds a paraphrase mutator. A centroid seed yields near-duplicate restatements; a spread seed yields a distinct trace family. Coverage comes from the edges.

References

Work	Bears on
Ankerst, M., Breunig, M. M., Kriegel, H.-P. and Sander, J. (1999). OPTICS: ordering points to identify the clustering structure. ACM SIGMOD Record 28, 49-60. DOI	Density-reachability ordering when cluster count and density are unknown
Campello, R. J. G. B., Moulavi, D. and Sander, J. (2013). Density-based clustering based on hierarchical density estimates. Advances in Knowledge Discovery and Data Mining (PAKDD), 160-172. DOI	Hierarchical density clustering with explicit noise, the basis of HDBSCAN