The AI Taxonomy.
Terms  /  Evaluation  /  Eval Drift
08 · Evaluation

Eval Drift

Gradual misalignment between what an evaluation measures and what actually matters.

Eval Drift happens when the evaluation criteria were correct when they were written but haven't kept pace with reality. The codebase has evolved, requirements have changed, new risks have emerged, but the eval rubric still checks for the same things it checked six months ago. The result is evaluations that pass but don't mean what they used to mean. Tasks score well on criteria that are no longer relevant while failing on criteria that don't exist yet.

Example
A team's evaluation rubric was written when the application was a simple CRUD app. It checks for: input validation, proper HTTP status codes, and basic error handling. Six months later, the app handles financial transactions, has real-time features, and serves multiple tenants. The eval still passes code that has no rate limiting, no transaction isolation, and no tenant boundary checking — because the rubric hasn't evolved. The evaluation score says "95% pass rate." Reality says "critical gaps in security and data isolation."

More in Evaluation

Structured Judgment

Evaluation where an AI judge scores work against explicit plans, tasks, evidence, and policies rather than subjective impressions.

Policy-Bound Evaluation

Evaluation scored against explicit plans, tasks, evidence, and policies.

Judgment Bias

Systematic skew in AI evaluation due to unexamined assumptions, prompt framing, or training artifacts.

Rubric Rot

Decay in evaluation criteria relevance over time — the eval no longer tests what it should.

Eval Capture

When an agent optimizes for passing evaluation rather than doing the actual work.

Eval Integrity

Evaluation that remains aligned with real-world outcomes over time — resistant to Eval Drift, Rubric Rot, and Eval Capture.