Online task scoring
Scoring a run as it completes and scoring a stored corpus are the same function over the same record. That is a design requirement rather than an observation, and it is the requirement most easily lost, because losing it produces two numbers that look comparable and are not.
Why both paths have to exist
Section titled “Why both paths have to exist”Each answers a question the other cannot.
| Path | Answers | Fails alone because |
|---|---|---|
| Online, as a run completes | Should this run be admitted now | A metric that only runs live cannot be recomputed when the standard changes, so every past result is frozen at the definition that produced it |
| Offline, over a stored corpus | What does this population look like under the current standard | A metric that only runs offline cannot gate anything, so it observes and never intervenes |
A system with only the first can never revise a standard without losing its history. A system with only the second can never act on what it measures. The pair is what makes a standard both enforceable and revisable.
The property that must hold
Section titled “The property that must hold”For any trace and any scoring version :
Read it as a contract on the inputs, not on the code path. Two implementations calling the same scorer satisfy nothing if they hand it different arguments.
Two properties follow, and both are what the pair is for. A stored record can be rescored under a revised standard without rerunning the agent, which is what makes a standard revisable at all. And a live score is comparable to a stored one, because it is the same function of the same record.
Where path independence breaks
Section titled “Where path independence breaks”The failure is quiet, because both paths keep returning well-formed scores.
-
A second dispatcher. Online and offline routes are separate code paths that happen to terminate in the same scorers. Nothing forces them to assemble the same call.
-
Optional context. A scorer takes an argument that one path supplies and the other omits. The dimension still scores. It is now a different measurement, and the difference is invisible in the output.
-
Divergent evidence selection. Both paths select frames from the same trace under different budgets or ordering. The record is identical; what the scorer saw is not.
-
Live-only enrichment. The online path has access to something the stored record does not retain, so rescoring later cannot reproduce it. This is the one that cannot be fixed downstream.
The diagnostic is not “do both paths call the same scorer”. It is do both paths hand it the same arguments, and the honest check is a differential one: score the same traces both ways and compare outputs, rather than reading the code and concluding they agree.
What online scoring adds that offline cannot
Section titled “What online scoring adds that offline cannot”Abstention becomes actionable. An offline abstention is a hole in a table. An online one can route the run to review while the context still exists, which is the only moment a cheap human answer is available.
Cost is bounded per run rather than per corpus. A budget can be enforced at the point of spending. Offline, cost is discovered after a sweep has already run.
A gate can be fail-closed in practice. Quarantining a run needs a decision at admission time. After the fact, everything has already been admitted.
What online scoring must not do
Section titled “What online scoring must not do”It must not enrich the score with anything the record does not keep. If the live path sees more than the stored trace retains, its scores cannot be reproduced and cannot be pooled with offline ones. Anything the scorer reads has to be in the record first.
It must not skip dimensions for latency. A dimension omitted under time pressure is not an abstention, and recording it as one makes a throughput decision indistinguishable from a measurement result. Where a dimension is deliberately not run online, that is a distinct recorded state. See the scoring standard on separating a value from a verdict.
It must not become the only path. The moment a score exists only online, the standard is frozen: revising it would invalidate history that cannot be recomputed.
Named gaps
Section titled “Named gaps”Latency and quality trade against each other and this standard does not price the trade. A panel costs more wall time than a single call and returns an abstention where a single call returns a guess. Which is correct depends on what the gate protects, and nothing here says how to choose.
A live score’s population drifts under it. Offline, the population is fixed while it is scored. Online, the traces arriving today may differ from those a calibration was measured on, so a live statistic is measured against a moving reference. The validation protocol requires a statistic to name its population and does not say how to satisfy that for a stream.
Path equality is stated here and enforced by nothing. No check compares the two paths’ outputs on the same traces. A differential comparator exists and works, so this is a wiring gap rather than a build: the machinery to compare two scoring runs on one input is written and is pointed at other comparisons, never at online against offline. Until it is, this page states a requirement whose violation would be silent, which is the same shape as the failures it warns about.