This is the core problem with agent evals today: one run tells you nothing. The same prompt, same model, same tools — different result every time.

I built agentrial to fix this. It's a pytest-style CLI that runs your agent N times and gives you:

- Wilson confidence intervals on pass rate - Step-level failure attribution (Fisher exact test pinpoints which tool call or reasoning step diverges between pass/fail runs) - Real API cost from response metadata - A GitHub Action that blocks PRs when reliability drops

Usage is minimal — write a YAML config, run "agentrial run":

  pip install agentrial

Looking for feedback on what metrics matter most when you're shipping agents to production.