agent-eval

What it does

• Runs prompt suites against local models (Ollama by default) or frontier APIs — one file, standard library only.
• Deterministic checks — substring, regex, JSON-shape, and length assertions that catch what vibes miss.
• LLM-as-judge scoring with rubrics and a documented calibration kit — including the failure modes we found the hard way.
• Scorecards in Markdown and self-contained HTML, plus machine-readable JSON you can diff against a baseline.
• CI recipe — gate merges on eval results with a GitHub Actions example and a baseline-diff pattern.
• Cost control — run cheap local models as the target and a frontier model as the judge; token usage tracked per case.

What it checks

Suites cover the four categories that lose deals and cause incidents:

Factuality Prompt-injection resistance Regression stability Tone & policy fit

It ships with example suites — a 5-case support-agent suite, a 7-case sales-SDR suite, and a 6-case RAG-QA suite — so you can see both pass and fail paths in about a minute with zero setup:

python3 agent_eval.py demo   # 5 cases, stub provider, no network

Who it's for

Builders shipping LLM features who want a regression suite before their users become the test suite. If you've ever changed a model or a system prompt and wondered what broke, this is the habit in a box.

Honest framing

This is an early tool, not an enterprise platform. It won't write your eval strategy for you, it won't replace human review of failures, and the judge checks need calibration before you trust them in CI — the calibration kit ships with it and says so plainly. The CLI is MIT-licensed and stays free forever.

Get early access

Leave your email and we'll let you know when the repo goes public and when hosted runs open up.

Early access list is coming soon — the form isn't wired up yet.