A framework for AI evaluation that generates, judges, and learns.
Evaluation can't be a static test suite anymore. Models change weekly. Agents execute multi-step traces. Output-level testing misses failures that happen inside the reasoning chain. Judgment Engineering is the methodology for evaluation that keeps up.
Traditional LLM evaluation is stuck in two modes:
Benchmarks provide shared standards, but they hit a saturation wall. Top models cluster above 90% on MMLU. Differences are statistically meaningless. Contamination is real. Scaffold gaming means the same model scores 62% with one setup and 70% with another.
Eval Engineering formalized evaluation as a discipline (a good step), but it still only scores outputs. It doesn't generate new tests. It doesn't improve itself. It can't see inside agent execution traces.
Neither mode handles what's actually happening in production: multi-agent systems that make autonomous decisions, accumulate errors across steps, and fail in ways that look fine from the outside.
Stage 1: Benchmarks Stage 2: Eval Engineering Stage 3: Judgment Engineering
----------------------- ------------------------- ----------------------------
Static test suites Custom eval pipelines Adaptive test generation
Public leaderboards Domain-specific criteria Agentic judges (trace-level)
One-time scoring Continuous loops Self-improving judge calibration
Output-level only Output-level only Full execution trace analysis
Judgment Engineering doesn't replace the first two stages. It builds on them.
Evaluation should create its own test cases. Not just run existing ones.
- Adaptive test scenarios generated from production failures
- Adversarial inputs targeting known model weaknesses
- Each evaluation cycle produces new tests, not just new scores
Evaluation should read the full story, not just the final answer.
- Agent-as-Judge: Judges that read execution traces, retrieve supporting files, and test code. >90% agreement with human experts (vs. ~70% for standard LLM-as-Judge).
- Multi-model panels: Multiple judges reduce individual model bias. Position bias, verbosity bias, and self-enhancement bias are documented problems.
- Natural language criteria: Evaluation rules written in plain language, not hard-coded logic. Domain experts define what "good" means without writing code.
- Evidence chains: Every verdict comes with reasoning and cited evidence from the trace.
Evaluation should get better over time without human intervention.
- Judge optimization (GEPA) closes the human-judge agreement gap by 10% average (up to 19% on specific tasks)
- Each cycle feeds back into test generation and judge calibration
- Judges calibrate against human expert decisions on high-stakes verdicts
- The evaluation system improves itself as a byproduct of doing its job
The refund agent example: A customer requests a refund. The agent retrieves the return policy, identifies three policy violations, then approves the refund anyway. The final output looks correct ("Refund approved"). Output-level testing would pass this. Trace-level evaluation catches the reasoning failure.
Multi-agent failures (from "Agents of Chaos" research):
- 124 internal emails containing SSNs and bank details disclosed to unauthorized agents
- Circular messaging loops burning 60,000+ tokens with no progress
- Agents reported tasks as complete while system state contradicted their reports
These failures are invisible to output-level evaluation. You can only catch them by reading the execution trace.
Not every evaluation needs a full agentic judge. The cost-effective approach routes by complexity:
80% Cheap deterministic checks
Format validation, keyword matching, length constraints
Cost: ~$0
15% Calibrated LLM judges
Semantic quality, domain accuracy, style compliance
Cost: ~$0.01-0.05 per evaluation
5% Full Agent-as-Judge
Complex traces, high-stakes decisions, anomaly investigation
Cost: ~$0.10-0.50 per evaluation
Total: ~$47 per 1,000 evaluations (multi-model panels) vs. $312 for human experts.
Use these to assess whether an evaluation system qualifies as Judgment Engineering:
-
Can it think? Does it generate adaptive tests, optimize its own judges, define criteria in natural language, and produce structured verdicts with evidence?
-
Can it see everything? Does it evaluate across vendors (no lock-in), use multi-model judge panels, and handle multiple modalities (text, image, code, voice)?
-
Does it keep watching? Is evaluation continuous and tied to model release cycles, not human schedules? Can it gate releases?
-
Does it work on your reality? Can it run custom datasets in private environments with domain-specific scoring? Public benchmarks cover maybe 10% of real use cases.
-
Who reviews the verdicts? Do humans adjudicate high-risk verdicts, recalibrate judges, and define escalation thresholds?
| Claim | Source |
|---|---|
| Traditional LLM-as-Judge: ~70% agreement with human experts | Judging LLM-as-a-Judge, 2023 |
| Agent-as-Judge: >90% agreement with human experts | Agent-as-Judge, 2024 |
| Judge optimization (GEPA): +10% average improvement | GEPA, 2025 |
| 64% of companies over $1B revenue lost >$1M to AI failures | Industry surveys, 2025 |
| Only 11% have agentic AI in production; 94% fail to scale past 25 systems | Enterprise AI deployment data, 2025 |
| EU AI Act enforcement: August 2026 (penalties up to 3% of global annual turnover) | EU regulation |
The full thesis with evidence, examples, and methodology:
manifesto/the-rise-of-judgment-engineering.md
Basic evaluation using the Stratix SDK:
from layerlens import PublicClient
client = PublicClient(api_key="your-api-key")
# List available benchmarks
benchmarks = client.list_benchmarks()
# Get evaluation results for a specific model
results = client.get_evaluation_results(
model_id="claude-opus-4-6",
benchmark_id="swe-bench-lite"
)
# Compare models across benchmarks
comparison = client.compare_models(
model_ids=["claude-opus-4-6", "gpt-4.1", "gemini-2.5-pro"],
benchmark_ids=["swe-bench-lite", "livecodebench", "arc-agi-2"]
)More examples in examples/.
judgment-engineering/
manifesto/
the-rise-of-judgment-engineering.md # Full thesis
methodology/
getting-started.md # How to apply JE principles
principles.md # Core principles expanded
examples/
basic-evaluation.py # Simple Stratix SDK usage
judge-pipeline.py # Multi-judge evaluation pipeline
trace-evaluation.py # Trace-level agent evaluation
README.md
Built by Jake Meany at LayerLens:
- JakeOS : AI-augmented operating system for solo marketing
- AI Marketing Skills : Production-tested skill library for AI agents
- DevPulse : Developer community research tool
- Awesome LLM Evaluation : Curated list of evaluation tools and benchmarks
MIT