Fleet and devices
A fleet is usually described as capacity. For measurement it is first a variance source, and one that is easy to leave out of the record because it feels like infrastructure rather than data.
Every score on this standard is a measurement taken in an environment. Change the environment and the same agent produces different traces, so a score without its environment is not reproducible and two scores from different environments are not comparable. That is the whole of why this page exists.
What the record has to carry
Section titled “What the record has to carry”A device profile is part of the evidence, not context around it. Without these, a later reader cannot tell an agent failure from an environment failure, which is the same reason architecture requires observed state rather than reported state.
| Recorded | Because without it |
|---|---|
| Device and platform identity | A regression cannot be attributed to the agent rather than a device swap |
| Screen geometry and pixel density | Any measure computed in pixel space is uninterpretable, see below |
| Application version under test | An app update changes the task, silently |
| Operating system version | Behavior and affordances shift under a fixed standard |
| Network condition class | Timeouts and partial loads read as agent hesitation |
The rule is the one the standard applies elsewhere: a value that is absent is recorded as absent. An unrecorded device profile is not a neutral omission, it is a silent claim that the environment did not matter.
Health is a measurement problem, not an operations problem
Section titled “Health is a measurement problem, not an operations problem”A degraded device produces traces that fail, and those failures land on the agent.
A stale application build, a full disk, a throttled radio, a device drifted out of its locale: each produces runs that stall, time out, or show the wrong screen. Every one scores as an agent failure under a standard that reads terminal state, because the terminal state genuinely does not show success. The measure behaves correctly and the conclusion is wrong.
That is the same error shape as capture degradation lowering task completion, and it has the same remedy: the environmental cause has to be in the record so the failure can be attributed. A fleet that reports health only to an operations dashboard cannot support that attribution afterwards.
Named consequence. A fleet-wide degradation and a genuine agent regression look identical in the aggregate. Both appear as a fall in task completion across a population. Separating them needs the device profile in the record, and no amount of statistical care recovers it later.
Scheduling entangles the task and device distributions
Section titled “Scheduling entangles the task and device distributions”How runs are assigned to devices is a measurement decision, not only a throughput one.
If harder tasks route to more capable devices, or if some devices fail more often and their runs are retried elsewhere, then task difficulty and device identity become correlated. A per-device comparison then measures the task mix, and a per-task comparison measures the device mix. Neither is wrong on its own terms and neither answers the question asked.
Two properties, both cheap at scheduling time and impossible to reconstruct afterwards:
- Assignment is recorded, including retries. A run that failed on one device and succeeded on another is two observations, not one, and the first is not noise.
- Assignment is independent of what is being compared. If the comparison is between agents, the device distribution must not differ between them. Randomizing assignment is the cheap way to buy that.
Fleet drift under a fixed standard
Section titled “Fleet drift under a fixed standard”The environment moves while the standard stays still. Operating systems update, applications ship new flows, a device is replaced by a different model. None of that changes the scoring contract and all of it changes what the contract is measuring.
The consequence for reporting over time is direct: a comparison across dates confounds agent change with environment change, and nothing in the standard separates them. The validation protocol requires a statistic to name its population, and for a fleet the population includes the fleet’s own state at the time. A statistic older than the environment it was measured in has quietly expired.
The concrete case: measures in pixel space
Section titled “The concrete case: measures in pixel space”Finding repeated-tap clusters means deciding when two taps are the same tap, which is a distance threshold. Where that threshold is expressed decides whether the category it feeds is a fleet measure or a device one.
A threshold in device pixels does not transfer. The same physical gesture spans more pixels on a denser screen, so it can cross a boundary on one device and not on another. A classifier threshold expressed in pixels is a per-device threshold wearing a fleet-wide label, and the category it assigns depends partly on which device ran.
The fix is normalization, not a better constant: carry coordinates in a normalized space and put the threshold in that space too, so a distance means the same fraction of a screen everywhere. Use a radius rather than a grid, since a grid splits two taps that fall either side of a cell edge no matter how close they are, which is the same error in a different coordinate.
A deployed detector already does both, thresholding on a fraction of a normalized screen diagonal rather than on pixels. The requirement is stated here because it holds for any measure in pixel space, not only that one, and nothing prevents the next threshold from being written in device pixels again.
Named gaps
Section titled “Named gaps”Fleet composition is absent from every published statistic. Nothing here requires a result to state which devices produced it, so two agreement numbers measured on different fleet mixes are presented as comparable. This is the population problem the validation protocol names, applied to hardware, and it is unaddressed.
This standard does not specify the health signals it requires. The argument above says they belong in the record, and this page does not say which ones, in what vocabulary, or at what granularity, so it is a requirement without a schema. One deployed fleet does define a fixed set on its device records, which shows the schema is writable; what is missing is a specified one that a result can be checked against, rather than one system’s choice read back as the standard.
Observation cost is unmeasured. Capturing state at every step changes a run’s timing, so a device under heavier instrumentation is a different environment from one under lighter. The size of that effect is unknown, which makes it a known unknown rather than a bounded one, the same gap architecture records from the other side.