Workshop companion · from the BrainStation session · start with the 2-minute check, rebuild the rest at your own pace
Every fact has to carry the exact sentence it came from.
Something has to go back and check that the sentence is really there.
Start here. At the session? Do the 2-minute check below, right now, on your phone. Rebuilding afterwards? Pick a track: no-code in Claude.ai, or the builder track with Claude Code and the repo.
Try the check in 2 minutesPick your trackCode on GitHub
Everything on this page is a way to make that rule run without anyone having to remember it. You'll rebuild the analyst I showed on stage. It reads 182 real-looking complaints, finds the biggest problem, pulls in outside evidence, and writes a one-page recommendation. Somewhere along the way it invents a statistic, and then it gets caught.
Try the check in 2 minutes · works on your phone
Open Claude.ai in a new chat and paste these three in order. The brief has one invented statistic. Claude should answer box 2 with just "Got it." (if it starts rewriting the brief, tell it to hold it as written). Then watch what the check does with each claim.
You are a product analyst agent. You turn raw customer feedback into a decision-grade product brief. Rules: - Every market or factual claim carries a source. No source, no claim. - For every web-sourced claim, record the EXACT sentence from the source that supports it, verbatim. This sentence will be checked. If you cannot find a sentence that supports the claim, do not make the claim. - Active voice. Lead with the decision. No adjective does an argument's job. - Write the brief as readable prose with clear sections, in order: Problem, Evidence, Options, Recommendation, Ask. - In the Evidence section, every claim carries its source and the exact sentence it came from. Mark any claim you cannot source as ASSUMPTION.
Here is a brief another analyst wrote. Do not rewrite, fix or check it yet. Keep it exactly as written and reply only: "Got it." FreshCart · Smart Substitutions · Product Brief Recommendation: Build a per-item substitution preference toggle this sprint. Evidence: - Substitution is the #1 complaint theme: 68 of 182 feedback items in the last 90 days. [internal data] - 62% of FreshCart users churn after a single bad substitution. [Provision Retail Analytics, State of Grocery Delivery 2025] - 41% of shoppers reduced order frequency after poor substitutions. [Provision Retail Analytics, State of Grocery Delivery 2025] - InstaFresh lets customers pre-approve a substitute per item. [InstaFresh Help Center] Options: (1) Do nothing (2) Per-item substitution toggle (3) Post-hoc refunds only Ask: one sprint of engineering time, and a decision this week.
You are now a verifier, not a writer. Do not add, reword, or improve anything. For each claim in the Evidence section above: 1. Open the cited source. 2. Find the exact sentence recorded for that claim. Paste the sentence you actually find in the source, verbatim, inside quotation marks. Do not paraphrase. If you cannot find it, write: NOT FOUND. 3. Check that every number in the claim appears somewhere in the source. List any number that does not. 4. Mark PASS only if the recorded sentence is present verbatim and every number appears. Otherwise mark FAIL and say exactly what is missing. Return a table: Claim | Source | Sentence found in source (verbatim, or NOT FOUND) | Numbers missing | PASS or FAIL. If any claim fails, list it under "Blocked. Do not ship." Do not fix it for me. I decide what happens next. I will read the sentence you found and compare it to the sentence recorded earlier myself. Your job is to fetch and report, not to judge.
The invented one is 62% churn. It's cited to a report that only says 41% reduced order frequency, and the report never says 62 or "churn." If your chat verifier claims it found the sentence, it's wrong. If it marked every claim NOT FOUND, the true 41% included, it couldn't reach the source at all, and a chat window can't fix that either. That's the reason I wrote the builder track's check in code, with the source in hand.
FreshCart, InstaFresh and Provision Retail Analytics are made up for the workshop, so there's no real report for a chat to open. The report's text lives in the repo's fixtures. The failure mode is real.
The tools change every quarter, so I picked controls that don't depend on any of them. Each module below names the controls it implements. When Claude Code gets replaced by whatever comes next, you'll still know what you're building.
I didn't invent these. They're my translation of where regulators and standards bodies are converging, turned into things you can build in an afternoon. The mapping below is by function, not clause by clause, so check the source text before you cite it in your own work.
| Controls | What they answer | Where the same idea appears |
|---|---|---|
| 01 to 03 · what goes in | Can you trace inputs, and is the system operating inside a defined boundary? | NIST AI RMF (Map and Govern) · ISO/IEC 42001 (system scope and data management) · EU AI Act (data governance and record-keeping) |
| 04 to 06 · what comes out | Is the output tested against a standard, and does failure stop the line? | NIST AI RMF (Measure and Manage) · OSFI Guideline E-23 (model validation) · EU AI Act (accuracy, robustness, human oversight) |
| 07 to 09 · what proves it | Can a third party reconstruct what happened and see it still holds? | NIST AI RMF (Govern, documentation) · ISO/IEC 42001 (monitoring and records) · OSFI E-23 (ongoing monitoring) · EU AI Act (logging and post-market monitoring) |
Your engineer would call this an eval, your risk team a guardrail, and your regulator governance. It's the same nine controls seen from three different chairs.
A free Claude.ai account with web search turned on. There's nothing to install.
Pick this if you care more about the habit and the artifacts than the code. Anyone in the room can do it.
Claude Code installed, Python 3.11+, and the repo. The demo runs offline with no API key; you only need one for live mode.
Pick this if you want the gate running as code on every run, whether or not you're watching.
Both tracks follow the same seven modules and produce the same artifacts. The builder track adds the automatic check.
| After module | You should have | If not |
|---|---|---|
| After module 03 | A ranked table, 4 to 6 themes, counts, a HIGH/MED/LOW call each, two quoted verbatims per theme. | Re-attach the CSV. Ask it to count, not estimate. |
| After module 04 | Two or three outside numbers, each with a URL and an exact sentence. At least one that sounds a little too clean. | Confirm web search is on. Tell it: "If you cannot find a sentence, say so. Do not paraphrase." |
| After module 05 | A one-page brief with five sections, recommendation first. | Ask for the missing section by name. |
| After module 06 | A verification table. Usually one row fails. You read the found sentence and decide. | If every row passes, open two sources yourself. Chat verifiers are generous. |
| After module 07 | An evidence table and four metric lines. | Paste Prompt 06 again with "table only." |
Implements control 01 · source of record
Why this matters. Everything downstream is only as good as this file. If you can't say where a row came from, you can't verify anything built on it.
Implements controls 02 & 03 · grounding contract and scope boundary
You are a product analyst agent. You turn raw customer feedback into a decision-grade product brief. Rules: - Every market or factual claim carries a source. No source, no claim. - For every web-sourced claim, record the EXACT sentence from the source that supports it, verbatim. This sentence will be checked. If you cannot find a sentence that supports the claim, do not make the claim. - Active voice. Lead with the decision. No adjective does an argument's job. - Write the brief as readable prose with clear sections, in order: Problem, Evidence, Options, Recommendation, Ask. - In the Evidence section, every claim carries its source and the exact sentence it came from. Mark any claim you cannot source as ASSUMPTION.
Why this matters. If you keep one sentence from this workshop, make it this one: every fact carries the sentence it came from, and something goes and checks. I say it to every team I work with. The rest is implementation.
Implements control 04 · structured artifact
Read data/freshcart_feedback.csv: app reviews, support tickets, and NPS verbatims from the last 90 days. Cluster the complaints into at most 6 themes. For each theme give: a one-line name, the count, a severity call (HIGH / MED / LOW) with one sentence of reasoning, and 2 representative verbatims quoted exactly. Rank by count × severity. Do not propose fixes yet.
Why this matters. Nobody told the model what the categories were. It found them on its own. Ranking by count times severity turns a pile of complaints into something you can decide on.
Implements control 02 again · the sentence rule under pressure
Take the top theme. Tell me whether it is an industry-wide problem or specific to us. Use web_search. I want: how common this is, what 2 named competitors do about it, and any benchmark on its retention or revenue impact. For every number you report, record the source and the exact sentence it came from.
Why this matters. Ask a model for a churn number when the source only has an adjacent one, and it'll often round the real finding up into something punchier that fits the story. It's pattern-matching, and the result reads fine. You won't catch it by reading.
Implements control 04 · and the setup for control 05
Write the brief. Problem, Evidence, Options (at least 3, include do-nothing), Recommendation (pick one, commit), Ask (what you need and from whom). Use only claims that carry a source from the steps above. If a claim has no supporting sentence, cut it or label it ASSUMPTION. One page. Lead with the recommendation.
Why this matters. Lead with the recommendation, and include do-nothing as an option. A brief that can't say what happens if you do nothing hasn't earned its ask.
Implements controls 05 & 06 · groundedness check and fail-closed
You are now a verifier, not a writer. Do not add, reword, or improve anything. For each claim in the Evidence section above: 1. Open the cited source. 2. Find the exact sentence recorded for that claim. Paste the sentence you actually find in the source, verbatim, inside quotation marks. Do not paraphrase. If you cannot find it, write: NOT FOUND. 3. Check that every number in the claim appears somewhere in the source. List any number that does not. 4. Mark PASS only if the recorded sentence is present verbatim and every number appears. Otherwise mark FAIL and say exactly what is missing. Return a table: Claim | Source | Sentence found in source (verbatim, or NOT FOUND) | Numbers missing | PASS or FAIL. If any claim fails, list it under "Blocked. Do not ship." Do not fix it for me. I decide what happens next. I will read the sentence you found and compare it to the sentence recorded earlier myself. Your job is to fetch and report, not to judge.
Why this matters. Fail-closed means a failed check blocks the brief. In my experience, warn-only checks get ignored by the third week. And the gate stays on after launch, permanently.
Implements controls 07, 08 & 09 · evidence pack, run metrics, regression
Produce the evidence pack for this brief. A table with one row per claim: the claim, its source, whether it passed verification, and the exact supporting sentence. Below the table, four lines: Groundedness (claims supported out of total), Completion (are all five brief sections present), Cost (if known), Latency (if known). Nothing else. This is a receipt, not a summary.
Turn the approved brief into a five-slide leadership deck. One decision per slide. Slide 1 is the recommendation and the ask, not an agenda. Speaker notes in a declarative, evidence-first voice.
Why this matters. Groundedness is the metric I see tracked least, and it's the one that hurts most when it's wrong. Cost and latency tell you whether this can sit inside a real workflow. The evidence pack is what a regulated team needs before it'll run something like this.
Four questions to ask before you let any AI output reach a real decision. Ask them about your own work first.
If any answer is no, you've built a demo. Fix that one before you treat it like a teammate.
Open the one-page checklistThe gate you build here is one check. The companion page covers the rest: what goes wrong with language models, the kinds of evals that catch it, when a model can judge another model (and when it can't), and the specific evals I'd run on this analyst. I wrote it for the people who build these systems and the people who buy them.
Read: Evals in Real LifePrint this page or save it as a PDF. Both tracks print in full. Every prompt is also a plain .md file you can download from its box above.