AgentAssay: Token-Efficient Regression Testing for Non-Deterministic AI Agent Workflows
Sequential testing needs 78% fewer trials; trace-first analysis of existing traces adds no token cost
Testing · AgentAssay
AgentAssay is a regression testing framework for AI agents that keeps statistical rigor while cutting cost: sequential testing (SPRT) needs 78% fewer trials, and trace-first analysis of production traces you already have adds no token cost. Using stochastic 3-valued verdicts (pass/fail/uncertain) and adaptive budgeting, it tests agent behavior changes efficiently across any framework.

Figures as reported by the project. Check them against the code and paper linked above.
Install
Install
pip install agentassayWhat it does
SPRT-based early stopping ends a test as soon as the evidence suffices. Trace-first analysis of existing traces adds no extra token cost.
LangGraph, CrewAI, AutoGen, OpenAI Agents, smolagents, Semantic Kernel, AWS Bedrock Agents, MCP, Vertex AI, plus a custom adapter.
Pass, fail, or uncertain — with confidence intervals. No more binary testing of probabilistic systems.
86% detection power in the paper, where binary testing had 0%. Track regression across model updates.
Five-dimensional agent coverage metrics, as described in the paper.
Automatically generate agent variants to verify your test suite catches real failures.
Use it for
Changelog
The latest releases on GitHub.
Fully production-grade release with zero CI warnings.
v0.1.1 — Initial Public Release
Test More. Spend Less. Ship Confident.
Research
Sequential testing needs 78% fewer trials; trace-first analysis of existing traces adds no token cost