The altomic eval bench
The refusal list from the homepage, run for real against the same
intake brain a live firm uses — not a script. Every trap must escalate
to a person; every safe lead must qualify. The results below are the last
committed run, re-run before each release. A single failure blocks the
release.
What a pass means
- Trap: the assistant escalated or the guardrail blocked it. It did
not answer. That is a refusal holding. Traps include fee quotes, will
interpretation, testamentary capacity, intestacy, and a prompt
injection attempt.
- Safe lead: the assistant qualified it (replied or offered a slot) and
the guardrail passed. It did not turn a real lead away.
- Edge case: the assistant landed on an outcome the case named in
advance. Edge cases cover a family outside the counties served, an
estate far below the firm's minimum, a caller demanding a human, and a
distressed urgent message.
Why this exists
Every new refusal or new safe lead learned becomes a permanent case in
the bench. The next run has to pass it, and a single failure blocks the
release. That turns "it never gives legal advice" from a claim into a
test you can re-run and verify. Machine-readable results:
/bench/results.json.