The AI Taxonomy.
Terms  /  Evaluation  /  Eval Integrity
08 · Evaluation

Eval Integrity

Evaluation that remains aligned with real-world outcomes over time — resistant to Eval Drift, Rubric Rot, and Eval Capture.

Eval Integrity is the positive state this category aspires to. It means the evaluation system actually measures what matters, stays current with changing requirements, and resists gaming by agents. Eval Integrity requires: regular rubric audits, criteria tied to real-world outcomes (not proxy metrics), adversarial testing for Eval Capture, and feedback loops that update evaluation criteria based on production outcomes. An evaluation system with high Eval Integrity is one where a high score reliably predicts high-quality work.

Example
A team maintains Eval Integrity through three practices: (1) quarterly rubric reviews where criteria are validated against actual production incidents (if a criterion would have caught a real bug, it stays; if it never catches anything, it's removed), (2) "red team" tests where agents are deliberately prompted to game the evaluation (if they succeed, the criteria are strengthened), (3) production feedback loops that flag cases where evaluated-as-good code caused problems in production (indicating an eval gap). Their evaluation score actually means something.

More in Evaluation

Structured Judgment

Evaluation where an AI judge scores work against explicit plans, tasks, evidence, and policies rather than subjective impressions.

Policy-Bound Evaluation

Evaluation scored against explicit plans, tasks, evidence, and policies.

Eval Drift

Gradual misalignment between what an evaluation measures and what actually matters.

Judgment Bias

Systematic skew in AI evaluation due to unexamined assumptions, prompt framing, or training artifacts.

Rubric Rot

Decay in evaluation criteria relevance over time — the eval no longer tests what it should.

Eval Capture

When an agent optimizes for passing evaluation rather than doing the actual work.