Skip to content

Build an AI Analyst You Can Trust · one-page checklist

The Trust Checklist

Four questions to ask before any AI output reaches a real decision. Ask them about your own work first.

  1. 1
    Can every fact name its source?

    "I researched it" doesn’t count. Name the actual source. If a number can’t say where it came from, it’s an assumption wearing a suit.

  2. 2
    Does something check that source automatically, or only a human on a good day?

    Humans skim, and experts skim faster. The check has to actually run every time, without anyone remembering to do it.

  3. 3
    When the check fails, does it block the document, or warn and get ignored?

    In my experience, warnings get ignored by the third week. A failed check should stop the document from shipping until a person decides what to do.

  4. 4
    Could you show someone the receipt six months from now?

    A record of every claim, its source, and whether it passed. When the VP asks where 62% came from, that’s what you want to hand them.

If any answer is no, you've built a demo that isn't ready to be a teammate yet. That's fine. Now you know which of the four to fix first.

From Build an AI Analyst You Can Trust, BrainStation Toronto · Khaled Zaky · khaledzaky.com. Engineers would call this an eval, risk teams a guardrail, and regulators governance. It's the same four questions either way.

If you want to turn these four questions into an engineering practice, read Evals in Real Life. It covers the kinds of evals, when to trust a model to judge, and the ones I'd run on a real analyst.

Back to the workshop