Build an AI Analyst You Can Trust · going deeper
Evals in real life.
The workshop showed one check on one document, which is the smallest honest version of the idea. This page covers the rest: what goes wrong with language models, the checks that catch it, when a model can grade another model, and the evals I'd run on the FreshCart analyst. The habit stays the same throughout. Every claim carries its source, and something goes and checks.
One check was the atom. Here is the molecule.
On stage, one deterministic check ran once against one brief and caught one invented statistic. That's the atom of the whole discipline: a claim, a source, and code that goes and verifies. A real system keeps the atom and scales it up. You check a set of cases instead of one, you run a suite of checks, and you run them at two moments: before you ship and while it's live.
The gate you saw is a single assertion. In production it becomes a versioned dataset of representative inputs with expected properties, a suite of independent checks scored per case, a CI job that fails the build on regression, and a sampling job that scores live traffic. The pass/fail is cheap. The dataset is what lasts, so treat it like code: review changes to it, and never let the model under test write its own test cases.
What actually goes wrong.
You can't check for failures you haven't named. These are the ones I see reach real decisions. Most of them are the predictable result of a system that completes patterns under pressure.
| Failure | In plain terms | What it takes to catch |
|---|---|---|
| Fabrication | A fact with no source at all, invented from nothing. | Less common than people expect, and the most alarming when it happens. Deterministic source checks catch it cleanly. |
| Citation drift | A real source, bent to a number or claim it never made. | What you saw on stage. It’s the dangerous one because it survives a careful read. |
| Overconfidence | A hedge-free answer to something genuinely uncertain. | Models don’t give you a confidence signal by default. Calibration and abstention need their own evals. |
| Omission | What it left out: the counter-evidence, the caveat, the risk. | The hardest to test, because you’re checking for an absence. |
| Nondeterminism | Same input, different answer tomorrow. | This is why you check every run. Pin what you can and measure variance on the rest. |
| Sycophancy | It agrees with the framing in your prompt. | Leading questions produce agreeable, wrong answers. Test with adversarial phrasings. |
| Silent regression | A model or prompt change quietly breaks what worked. | It’s why evals belong in CI. A suite that runs on every change catches it before your users do. |
| Prompt injection | Text in the input hijacks the system’s instructions. | This one’s an input-side threat, so guardrails handle it rather than offline evals. |
Two more that bite in production and rarely show up in demos. Goodhart: once a metric becomes the target, the system optimizes the metric instead of the goal (a groundedness score climbs while the answers get evasive). Eval-set contamination: test cases leak into prompts, fine-tunes, or few-shot examples, and the suite goes green for the wrong reason. Rotate held-out cases and keep a private set the pipeline never sees.
What you can check.
"Eval" covers a family of questions, and each one gets answered a different way. You rarely need all of them. Pick the two or three that match how your system fails.
| Kind of eval | The question it answers | How it's usually done |
|---|---|---|
| Groundedness / faithfulness | Can every claim name the exact sentence it came from? | Shown on stage: exact quote plus number presence. For open-ended RAG, entailment (NLI) or a judge with a rubric. |
| Correctness / task success | Did it actually do the job that was asked? | Needs a reference or a checkable outcome. The clearest signal when you can define it. |
| Completeness / schema | Is every required part present, in the required shape? | Cheap and deterministic. Validate the structure before you check anything semantic. |
| Relevance / helpfulness | Is the answer on-point and useful, not just true? | Fuzzy. Usually a rubric with a judge or a human, and it’s where I see people over-trust judges most. |
| Safety | Toxicity, PII leakage, resistance to being manipulated. | Classifiers + policy checks; overlaps with runtime guardrails. |
| Robustness / adversarial | Does it hold up under hostile or edge-case inputs? | A red-team set of nasty cases, run like any other regression suite. |
| Regression | Does the same set still pass after every change? | More a practice than a metric. The saved set is the asset, and pass/fail is the gate. |
How you check, and where each one fails.
I group the ways to grade an answer into four. The skill is matching the method to the question, and knowing how the method itself fails.
1 · Deterministic code
This is the check you built: fetch the source, look for the sentence, recompute the number. It's exact and cheap, and you can't talk it out of a verdict. Fails when what you care about can't be reduced to a rule, like tone, nuance, or "is this actually helpful."
2 · Reference-based
Compare the answer to a known-good one. When a gold answer exists, the signal is clear. Fails when there's no single right answer, or when a correct response is worded differently from the reference.
3 · A model as the judge
One model grades another's output against a rubric. It handles nuance that code can't touch. Fails when you trust it blindly. Judges have real biases, and they can be wrong in the same direction as the writer.
Known biases to design around: position (favors the first or last option), verbosity (favors longer answers), self-preference (favors its own style or family), and correlated error (misses exactly what the generator missed). Mitigations: score against an explicit rubric instead of vibes, calibrate the judge against human labels before you trust its numbers, randomize order and run pairwise, and use judges to rank and triage rather than to gate the safety-critical path. I keep the check that matters most deterministic, which is why the stage gate was code.
4 · Human review
A person judges, and the other three methods are approximations of this. Fails when you need it at scale or on every run, because humans skim and experts skim faster. Use people to calibrate the cheap checks rather than to be the check.
Two moments: before you ship, and while it's live.
Run the suite against your saved set every time anything changes: a prompt, a model, a tool. A red suite blocks the release. This is where the stage gate lives.
Sample real traffic and score it in production. Watch groundedness over time, not just at launch. They're different jobs. Passing offline doesn't mean it holds up on inputs you never imagined.
The step people skip: canary a model upgrade against the frozen suite before you cut over. A newer model can score better on general benchmarks and still quietly regress your groundedness, so diff the suite before you switch. In production, alert on the trend rather than a single bad run.
Guardrails are not evals.
I see people treat these as one thing and then wonder why one layer isn't enough. An eval is the crash test: it runs offline and measures quality against a set. A guardrail is the seatbelt: it runs at request time on the live input and output, and it can block or rewrite before the user sees anything. You need both. The crash test tells you it's safe to ship, and the seatbelt covers the ride you didn't test for.
Runtime guardrails: input filters (prompt-injection and jailbreak detection, PII scrubbing), output filters (toxicity, PII, policy, groundedness-at-inference on retrieved context), and refusal/abstention when confidence is low. Evals measure the system in aggregate, and guardrails defend a single request. A groundedness check can live in both: offline as a suite metric, online as a gate on the response.
The evals you'd actually run on this analyst.
Here's what I'd run on the FreshCart brief. You've already watched two of them run.
- Groundednessshown on stage
Each web claim’s exact quote has to appear in the cited source. Deterministic.
- Numeric recomputeshown on stage
Internal statistics get re-derived from the CSV instead of trusted as stated.
- Completeness / schema
All five brief sections present; every claim carries a source field.
- Retrieval quality
The cited sources are real, reachable, and actually on topic.
- Scope
It read only the feedback file and the sources it cited. Nothing it wasn’t handed.
- Regression set
A handful of briefs with known-good outcomes, re-run on every prompt or model change.
What the gate does not catch.
This is the part I most want you to take away. Even a perfect groundedness gate only catches a fabricated citation. It won't catch:
- A real source bent to the wrong conclusion, where every sentence checks out and the argument is still false.
- Omitted counter-evidence, like the study it didn't cite because it undercut the recommendation.
- A reasoning error: sound facts, invalid inference.
- The question it should have asked and didn't.
No single eval covers all of these. That's why you run a suite, why you add human judgment for what code can't reach, and why a person still signs off. The gate takes away the failure people are worst at catching (the confident, well-sourced, wrong number), so the person signing can spend their judgment where only judgment works.
How to interrogate any output.
These are the checklist's four questions turned into six moves you can make on a single answer, before you build any of this. Do them by hand first, then automate the ones that repeat.
- 1Make it name the sentence.
For every fact, ask for the exact sentence from the source, word for word. If it can’t produce one, treat the fact as an assumption.
- 2Recompute the number yourself.
Any internal figure should be re-derivable from the data, so do the count yourself instead of trusting the stated total.
- 3Ask what would make it wrong.
A system that can’t state its own failure conditions hasn’t reasoned about them. Make it argue against itself.
- 4Check the sources are real.
Open two and confirm each page exists and says what was claimed. Plausible-looking citations are the easiest thing to fabricate.
- 5Run it twice and compare.
Same prompt, fresh session. Where the two answers disagree is where the uncertainty is hiding.
- 6Look for what’s missing.
Ask for the counter-argument, the caveat, and the do-nothing option by name. Omissions stay invisible until you do.
The habit stays the same as the toolkit grows.
Every technique on this page runs the same rule (every claim carries its source, and something goes and checks) at more scale, against more failure modes, and at more moments. My advice is to start with the one deterministic check that catches your worst failure, and add the next one when a real miss earns it. Reach for a model-as-judge only where code can't do the job, and never for the decision that matters most.
Tooling for all of this exists (eval harnesses, tracing and observability, guardrail runtimes), and it changes every year. The principle outlasts any of it: name your failures, save your set, fail closed, and keep the receipt.