The AI Taxonomy.
Terms  /  Evaluation  /  Structured Judgment
08 · Evaluation

Structured Judgment

Evaluation where an AI judge scores work against explicit plans, tasks, evidence, and policies rather than subjective impressions.

Structured Judgment replaces "does this look good?" with "does this meet the criteria?" It works by defining evaluation criteria upfront — tied to the plan, the requirements, and the policies — and then scoring the agent's output against those criteria systematically. Each criterion gets a score, evidence is cited, and the overall assessment is grounded in specifics rather than vibes. This approach is reproducible, auditable, and resistant to the model's tendency to produce output that sounds impressive but doesn't actually meet requirements.

Example
An agent completes a feature implementation. Instead of asking "is this good?", Structured Judgment evaluates: (1) Does the implementation match all 5 acceptance criteria in the task? Score: 4/5 — missing input validation. (2) Does the code follow the team's style guide? Score: 8/8 checks passing. (3) Are tests present and passing? Score: 3/4 — missing edge case for empty input. (4) Is the documentation updated? Score: 0/1 — not done. Overall: 15/18, with specific deficiencies identified. Compare this to "the code looks clean" — which tells you nothing about whether it actually works.

More in Evaluation

Policy-Bound Evaluation

Evaluation scored against explicit plans, tasks, evidence, and policies.

Eval Drift

Gradual misalignment between what an evaluation measures and what actually matters.

Judgment Bias

Systematic skew in AI evaluation due to unexamined assumptions, prompt framing, or training artifacts.

Rubric Rot

Decay in evaluation criteria relevance over time — the eval no longer tests what it should.

Eval Capture

When an agent optimizes for passing evaluation rather than doing the actual work.

Eval Integrity

Evaluation that remains aligned with real-world outcomes over time — resistant to Eval Drift, Rubric Rot, and Eval Capture.