The Test

An evaluation set is a typed, versioned object, not a folder of traces. Its card explains the set. Its records carry provenance. A lint gate checks both before the set is used.

Contract

An evaluation result is meaningful only when a reader can recover the exact set it came from. The contract therefore binds a card, a schema, and immutable records into one versioned object. It says what the object is, not where it is stored or how a vendor transports it.

Definition

An evaluation set at version $v$ is $S_v = (C_v, \Sigma_v, R_v)$: its card, schema, and immutable records. A scoring run records $v$; it never refers to a moving set label.

Reading. The card lets a reader understand the set. The schema lets software read it. The records supply the evidence. Change any of the three and mint a new version.

Figure 1.Structural contract for one evaluation set. The figure shows ownership and pinning, not a storage architecture or a measured result.

The labels on a record are produced under the LLM Judge calibration protocol. The card declares the split tier and provenance fields that make later leakage and lineage checks possible. The scoring path is separate: this page defines the test object it consumes.

Card and records

The card says what the set is for. The schema says which fields a record may contain. Each record carries enough lineage to explain where it came from and which choices could have changed it.

Object	Must state	Why it matters
Card	name, owner, purpose, source, license, split tier, and related sets	A reader can identify the set and its intended use.
Schema	required fields, allowed values, and expected evidence	A loader can reject malformed records.
Record	unique, never-reused record identifier; source event; generating model and prompt version; labeler; and raw-versus-edited flag	A result can be traced to the evidence and labeling procedure that produced it.

The schema exists before records are accepted. Records move from captured, to typed against it, to labeled, to promoted into a set. A blank expected field is refused. A value a person edited is flagged as edited, never silently mixed with raw capture. Displayed fields may not contain raw nulls, serialized blobs, local paths, or unmasked personal data.

The card declares one split tier: public, private, or held-out. Related sets link explicitly. A set is immutable once versioned: any change to its card, schema, or records mints a new version, and the old version remains addressable. The card declares a versioned hash profile: digest algorithm, canonical UTF-8 serialization, Unicode normalization form, canonical field order, and record order by the never-reused record identifier. The version is the digest of the card, schema, and records under that profile, so every scoring run can pin exactly what it judged.

The card follows dataset-documentation practice (Gebru et al. 2021) and exports shared metadata so sets can interoperate with public tooling (Akhtar et al. 2024).

Lint gate

The lint gate checks the contract before a set reaches a scorer. It runs on every set change in continuous integration and can run locally on demand. Conforming loaders also verify stored row hashes on the ordinary load path: a set cannot be read, joined, or reported until that check passes. These checks are structural. They pass or fail; this page defines no estimator.

Check family	Gate
Naming	Require a content-describing name and named owner. Reject duplicates, model configurations, operator instructions, and personal names.
Structure	Require card fields and schema-conformant records. Reject raw nulls, serialized blobs, and absolute paths in displayed fields.
Safety	Detect and mask personal data by default, or refuse the record.
Integrity	Verify stored row hashes on ordinary load. Scan held-out versus training overlap by hash and near duplicate. Require explicit links between sets that share content.
Attribution	Require each label file to bind its schema, rubric, and taxonomy versions at labeling time.
Reporting	Require a denominator and side-by-side judged, total, and pending counts for every reported number. Reject a lone success rate.

Failure	Contract response
Lint failure	Block the change.
Personal data detected	Mask it by default, or refuse the record.
Held-out and training overlap	Fail the set and report the overlapping record.
Missing specification binding	Refuse the label file. Do not infer a version from its name.

Limit. The two controls below are requirements, not current guarantees. Until both are enforced, a readable set is not necessarily a conforming set.

Conformance gap: labeled files carry no attribution to the specifications they were labeled under

Current labeled files omit their schema, rubric, and taxonomy bindings. They can be read and joined, but they do not reveal the definitions under which their labels were created.

Conformance gap: the lock check is opt-in

Rows have stored hashes, but ordinary loading does not verify them. A set can be reported without evidence that it remains the set that was locked.

Schema evolution remains open: a breaking schema change may mint a schema version, a set version, or both, and migration of old sets is not specified. The near-duplicate scan also has no validated default window or threshold. Those choices need evidence before they become release rules.

References

Work	Bears on
Gebru, T. et al. (2021). Datasheets for datasets. Communications of the ACM 64(12), 86-92.	The documentation practice the card follows.
Akhtar, M. et al. (2024). Croissant: a metadata format for ML-ready datasets. arXiv:2403.19546.	The shared metadata format a card exports.