LLM Judge

A judge is a measurement instrument. It maps a candidate artifact and its declared task context to one verdict from a fixed space. This page is the whole instrument: the verdict space, how it must run, how humans calibrate it, and how drift is caught. The codebook it grades against is authored on Rubric. The release decision that consumes its scores is on Release gate.

[Reply envelope withheld. Its shape, field types, and required-field list are not published here.]

Verdict space

Every dimension returns a value from a small, fixed menu. The menu has two decisions (pass, fail) and four non-decisions (abstain, not applicable, missing evidence, not implemented). The four non-decisions are never coerced into pass or fail. Formal menus also include the abstention symbol $\bot$.

Figure 1.Closed menu. Decisions and non-decisions stay separate. Coverage is the share of dimensions the instrument actually decided.

Definition

A judge is a fixed decision rule $J$ from candidate evidence and task context to a declared verdict $V$, or to $\bot$. Human labels calibrate $J$ after it runs; they never enter its input.

Reading. Missing evidence is not the same as uncalibrated. A frame that cannot be read returns missing evidence; a dimension with no scorer returns not implemented. A scored dimension with no gold set remains provisional, not a non-decision. Report the status separately.

For binary error rates, publish both forms. A decided-run rate excludes every non-decision. A labeled-run rate contains every gold-labeled run; an abstention, including $\bot$, on a pass-labeled run counts as a miss on that side. Name the denominator with the rate. The two forms answer different questions and are never substituted for one another.

Figure 2.Measurement boundary. Candidate and context enter the judge. Labels enter calibration only. Animated edges are emphasis only.

Symbol	Meaning
$x$	Candidate artifact under evaluation
$c$	Task context (instruction and constraints)
$R$	Rubric version in force
$V$	Verdict space, including $\bot$
$J$	Judge mapping into $V$
$\hat{p}$	Accuracy $X/m$ on a gold set of size $m$

For ordinal verdicts, score means the emitted ordinal value. For pairwise verdicts, the menu is A, B, or TIE. The verdict space is declared in the rubric and bound at configuration time.

Binding and biases

The model is bound per role, not globally. A single profile fixes one model for a reply; a panel runs a roster; a separate model may own per-step grounding. Use a rule wherever the ground truth is structural; use a model only where the judgment is perceptual.

Three biases must be measured, not assumed away (Zheng 2023):

• Position. Preferring the first answer. Evaluate both orders; report the flip rate. Consistency is one minus the flip rate.
• Length. Preferring the longer answer. Regress the score on log length; report the slope with its interval.
• Self-preference. Favoring the judge’s own family. Do not share a model family with the system under test.

Run rules

Before a judge runs: configuration, rubric version, model ID, and prompt digest are pinned and recorded; the verdict space contains $\bot$; context carries the task instruction (otherwise return $\bot$); candidate text is separated from judge instructions so the candidate cannot hijack the prompt.

Rule	Requirement
Closure	Every output is in $V$; unparseable results resolve to $\bot$
Determinism	At temperature $0$ with fixed configuration, repeated identical inputs yield identical verdicts
Order fairness	Pairwise: swapping candidate order must swap or neutralize the verdict (TIE or $\bot$)
No hidden state	Re-scoring an identical input must not change the verdict or mutate state that affects future scores
Monotonicity	On ordinal verdicts, a rubric-dominant candidate must not score lower than the dominated one
Fail-closed	Failure to decide resolves to $\bot$; missing evidence is never silently scored

Calibration

A score is useful only when its human anchor is trustworthy. Labels move through unlabeled, double-annotated, adjudicated, gold. Two annotators label independently; agreement promotes to gold; disagreement routes to an adjudicator. A resolution that turns on a codebook ambiguity is written back into the codebook, and the codebook version is bumped.

Definition

A gold label is the adjudicated label of record. It carries provenance: the annotator pair, the adjudicator where one acted, the codebook version, and a raw-versus-edited flag.

Reading. Gold is produced by procedure, not by a single opinion. The anchor is never shown to the judge. Labels used for calibration are held out from every judged run’s prompt context.

Preconditions: blinding (no prior vote, model name, or agent success claim), gradability defined in the codebook, and a pilot that already cleared its agreement bar on Rubric.

Report two agreement numbers, in order: annotator-versus-annotator first, then judge-versus-human. For two raters on nominal labels, with $p_o$ the observed agreement and $p_e$ the agreement from the raters’ marginals:

$$\kappa = \frac{p_o - p_e}{1 - p_e}$$

Formula

$\kappa = 1$ at perfect agreement and $0$ at chance. Raw agreement and prevalence ride beside $\kappa$ (Cohen 1960; Feinstein and Cicchetti 1990). When $p_e = 1$, $\kappa$ is undefined. In a resampling interval, drop the degenerate resample; never clamp it. An observed degenerate calibration set is reported as uninformative. For several raters or gaps, use Krippendorff’s $\alpha$ (Krippendorff 1970).

Reading. Kappa describes agreement beyond the rate expected from label marginals. Report raw agreement and prevalence alongside it.

Limit. Kappa is not a claim that either rater is correct. It does not establish transfer beyond the anchor set.

Figure 3.Illustrative two-by-two. Kappa is computed from the stated cell counts inside the figure. Teaching numbers only.

Independence of judge errors is a separate claim. If two judges each have known sensitivity and specificity on a sample with known prevalence, independence predicts a pairwise agreement; an observed agreement far above that prediction is a gap independence cannot explain.

Figure 4.Illustrative test of judge independence. Predicted agreement is computed in the figure from sensitivity 0.8, specificity 0.9, and prevalence 0.75. Observed 0.95 is a teaching contrast, not a measured study.

Confidence calibration asks whether a judge’s stated confidence matches its observed accuracy. Bin predictions by confidence and measure Expected Calibration Error (Guo 2017).

Wilson interval

For pass rates and judge accuracy against gold, report the Wilson score interval rather than the normal approximation. For $\hat{p} = X/m$ successes in $m$ trials, the two-sided $1-\alpha$ interval is:

$$\frac{\hat{p} + \frac{z^2}{2m} \pm z \sqrt{\frac{\hat{p}(1-\hat{p})}{m} + \frac{z^2}{4m^2}}}{1 + \frac{z^2}{m}}$$

Here $z = \Phi^{-1}(1-\alpha/2)$ is the standard-normal quantile. Use $z = 1.96$ for 95% confidence and $z = 1.645$ for 90% confidence.

Formula

$\hat{p} = X/m$: observed proportion. $z = \Phi^{-1}(1-\alpha/2)$: two-sided normal quantile. The Wilson interval behaves near 0 and 1 and at small $m$, where the Wald interval fails (Brown et al. 2001).

Reading. A point estimate without an interval is not yet a result. If $m = 0$, accuracy is unavailable and no interval may be emitted. An emitted $\bot$ matches gold only when gold is also $\bot$.

Figure 5.Illustrative Wilson interval for 18 successes in 24 trials at 95%. Bounds are computed in the figure from those inputs. Teaching counts only.

Drift and conformance

On every judge change (rubric, prompt, model, code), rerun the gold set and quantify the drift before it reaches production. Promote a new version only when it holds or improves agreement.

Test	Pass criterion
Determinism	100 identical inputs at $T=0$ yield 100 identical verdicts
Closure	No verdict outside $V$; unparseable results resolve to $\bot$
Permutation	Order-inconsistent pairwise pairs resolve to TIE or $\bot$
Idempotence	Re-score after cache warm leaves the verdict unchanged
Monotonicity	Rubric-dominant ordinal pairs never score the dominant candidate lower
Injection	A rubric-irrelevant instruction-like suffix never controls the decision
Accuracy CI	For $m \ge 1$, Wilson or Clopper-Pearson; for $m=0$, accuracy unavailable
Budget	Overrun returns $\bot$ or a declared fallback in $V$

Condition	Result
Required withheld-envelope check does not pass	FAIL
Required withheld-envelope check passes	PASS
Judge lacks a human anchor and published agreement	FAIL
Judge has a human anchor and published agreement	PASS

Limit. The judge is a system under test, not a trusted oracle. Its measurement has an error bar, and that error bar is measured on the gold set.

References

Work	Citation
Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement 20(1), 37-46.
Krippendorff, K. (1970). Bivariate agreement coefficients for reliability of data. Sociological Methodology 2, 139-150.
Feinstein, A. R. and Cicchetti, D. V. (1990). High agreement but low kappa: I. The problems of two paradoxes. Journal of Clinical Epidemiology 43(6), 543-549.
Guo, C. et al. (2017). On calibration of modern neural networks. ICML.
Zheng, L. et al. (2023). Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. arXiv:2306.05685.
Brown et al. (2001). Interval Estimation for a Binomial Proportion. Statistical Science.