Testing
AgentAssay
Stochastic testing for agents: verdicts over repeated runs that account for randomness, at a fraction of the tokens a naive repeat-and-compare spends.
CodeAGPL-3.0
Topic hub · Evaluation
Testing agents that never answer the same way twice.
An agent's output legitimately changes from run to run, so a golden-file diff fails on correct runs and passes broken ones. Testing has to judge the distribution of runs, at a token cost a team can pay.
Start here
Read it in this order.
Your progress is saved in this browser.
Reading path · 0 of 5 read
- PaperAgentAssay: Token-Efficient Regression Testing for Non-Deterministic AI Agent Workflows
- ToolAgentAssay: Token-Efficient Agent Testing
- PostThe Cheap Model Is Not Cheap Until It Finishes the Trace: Gemini 3.8 Flash & Meta Muse Spark 1.3 in Production
- PostWorld Models: Why the AI Race Is Moving Beyond Next-Token Prediction
- PostThe Reasoning Trap: Why Smarter AI Agents Hallucinate More
Tools
What to run.
Research
The papers.
AgentAssay: Token-Efficient Regression Testing for Non-Deterministic AI Agent Workflows
Sequential testing needs 78% fewer trials; trace-first analysis of existing traces adds no token cost
Glossary
Terms you'll meet.
- Stochastic verdict
A test result reached over repeated runs of a non-deterministic agent, judging the distribution of outcomes instead of a single sample.
- Non-deterministic regression testing
Checking that a change did not make an agent worse when its output differs every run, so comparing against a golden file is not possible.
Writing
Posts on evaluation.
Field reports
The Cheap Model Is Not Cheap Until It Finishes the Trace: Gemini 3.8 Flash & Meta Muse Spark 1.3 in Production
Google's Gemini 3.8 Flash and Meta's Muse Spark 1.3 promise frontier intelligence at budget rates. But across Hermes, OmniRoute, and OpenRouter, token price is only 20% of your bill. Here is why model portfolio routing beats leaderboard hype.
15 min read
Field reports
World Models: Why the AI Race Is Moving Beyond Next-Token Prediction
LLMs are not dead. But the next AI race is about predicting what changes after an action. Cosmos 3, Qwen-Robot, the taxi-map trap, and the reliability test that matters.
16 min read
Research explained
The Reasoning Trap: Why Smarter AI Agents Hallucinate More
An ACL 2026 paper just proved RL-trained reasoning causally amplifies tool hallucination. Here's the mechanism, the math, and what AI Reliability Engineering does about it.
13 min read
Field reports
The Pass^k Wall: One Failure Mode Behind AI's Quietly Disastrous Week
Anthropic missed three regressions. Uber burned its 2026 AI budget. 300k Ollama servers leaked memory. Princeton paused its leaderboard. Five headlines, one engineering failure: reliability under accumulated state. The metric that exposes it, three Monday-morning fixes, and the runtime contract framework that gates it.
12 min read
Field reports
GPT-5.5 vs Claude vs Gemini: The Avengers Problem Nobody Talks About
I ran GPT-5.5, Claude Opus 4.6, and Gemini 3.1 Pro through real benchmarks. Iron Man, Captain America, and Thor all showed up. Nobody won. Here's why that's actually the point.
8 min read
How we built it
How Adversarial Judge Pipelines Make AI Agents Trustworthy
Most AI agent frameworks skip output quality entirely. We built a 2-round adversarial judge pipeline with multi-model consensus, anti-fabrication verification, and configurable profiles — and tested the same principle by having 7 independent AI auditors evaluate our own codebase.
10 min read