Your LLM Judge Should Earn the First Call

TL;DR: I thought Jev was hype, then my own untuned safety testing returned ROC AUC between 0.96 and 0.99. The public phishing data tells a different story, with Jev losing to Haiku on another task. The lesson is to qualify each decision, then start with the least expensive evaluator that clears the bar and escalate from there. That makes the verification budget something you allocate within a workflow, not just when you design it.

A hosting-list rule scores 91.6% accuracy on a synthetic-email phishing benchmark. Jev’s broad verdict scores 62.6%. Claude Haiku 4.5 scores 81.3%. Both models lose to the simpler check.

I am writing about a new model, and the baseline is already giving me homework.

I rarely find these things first. Most of what I write starts when someone pokes me: a link, a question I cannot answer, a “have you seen this?” Then I go read until I can hold my own in the conversation. It is one of the best parts of working with smart people.

Karsten Economou starts this one. He sends me a link to Jev and says we should look at it.

TypeSafe has just come out of stealth. Jev launched on September 15. I fill out the waitlist form.

Then I drive home and do what I do most days now. My commute is about an hour, and I used to fill it with podcasts. These days I talk to Grok instead, mostly because it comes with my Tesla. It has turned time in traffic into one of the most productive hours of my day.

Tonight I ask it to explain what I have just signed up for. How does a model that does not generate text differ from an LLM? Where would a typed decision help?

So a generative model spends my commute explaining a non-generative one.

I get home with the start of a conclusion and no data behind it.

My assumption is that this is hype.

Then I run it against safety checks I already know well: prompt harm, response harm, and refusal detection. I score the results against human annotations. I use a one-shot prompt, with no tuning and no prompt experimentation. One-shot means I gave the model a single worked example of the task inside the prompt, wrote that prompt once, and did not iterate on it.

ROC AUC lands between 0.96 and 0.99 across the three checks.

ROC AUC measures how well a model ranks. Pick one example that should be flagged and one that should not, at random. ROC AUC is the chance the model scores the flagged one higher. 0.5 is a coin flip and 1.0 is perfect, so 0.96 to 0.99 means the model almost always ranked the right example higher. It still does not tell me where to set a blocking threshold in production.

The combined request returns in roughly 340 milliseconds, plus or minus 100 milliseconds. The checks cover both input and output and are answered in parallel in one request.

Adding another safety check adds input tokens. TypeSafe’s published pricing leaves output unbilled. For the checks I am adding, the marginal inference cost is near zero.

These are my own measurements of one configuration, not a benchmark.

On my checks, there is something real behind the hype. On the public phishing benchmark, Jev’s broad verdict loses to Haiku.

The question I bring into the weekend is whether the heavy LLM judge still needs to be the default.

In Your Judge Is Not an Independent Reviewer, I had to slow down and learn what the statistics actually established before carrying the argument further.

This time, I have encouraging results of my own and public results pointing the other way. Qualifying the decision, rather than approving the model in the abstract, is what makes sense of both.

The Week Moved Faster Than My Draft

Later in the week, Gaurav pokes me too. He sends jevals, an evaluation library built around the same typed-decision interface. Gaurav and I have a habit of arriving at similar questions from different directions. Someone had already written code for part of the answer.

September 15: TypeSafe exits stealth and launches Jev.

September 17: LangChain publishes a Jev integration, including a component that can block an agent’s proposed action (a tool call) before it runs.

September 17: Kev’s repository appears with a compatible decision interface.

September 19: Vercel publishes its Jev and AI SDK guide.

September 19: Laya, an open alternative to Jev, can now run locally on Apple silicon laptops.

September 20: LangChain publishes its Jev-as-an-evaluator experiment.

September 20: jevals publishes its first version, combining evaluation definitions with runtime gates.

By this point, it is a Jev maxxing weekend.

I expect a lot more Jev talk in the coming weeks. I am watching something wider than one model. Independent projects copied the request format within days, and an evaluation library already builds on it. That looks like a category forming.

jevals also arrives at arguments I recognize.

Its README says “the gate in production enforces exactly what you measured offline”. That is the implementation direction I wanted in June’s distinction between evaluations and guardrails: reuse the judgment, while making its different roles explicit.

It also says “don’t let the classifier become the authorizer”. That connects directly to the decision-engine argument.

The authors explicitly retain LLM judges for evaluations requiring substantial reasoning or written critique.

Openlayer sells evaluation tooling, so a vendor building in this direction is both a meaningful product signal and a commercial interest.

jevals is still alpha, with a short initial commit history. I am reading it as a design to learn from, not a production-ready tool.

Did Jev Already Lose to Open Weights?

Open weights means the model files are published, so anyone can download the model, run it on their own hardware, and retrain it. Jev is closed. You can only reach it through TypeSafe’s API.

I also try Laya and find it weak out of the box. On the typed-decisions benchmark, its authors report 0.362 zero-shot accuracy, against a 0.318 random baseline and a 0.461 majority-class baseline. Always guessing the most common answer beats it. They describe the base model as something to specialize.

Zero-shot means using the model as it ships, without training it on examples from your own task. Fine-tuning means adapting it with examples from your own data. For calibration error, lower is better.

That fine-tunability is the part that interests me. Their specialist checkpoint reports 0.766 accuracy against a published Jev result of 0.727 on four synthetic workflows it was fine-tuned for. Those are author-reported results, unreproduced here. The authors did not run Jev themselves, and the prompts and sample sizes differ. The specialist’s calibration error is also higher: 0.213 against Jev’s quoted 0.144.

On Kev’s new-source development suite, the authors report 0.796 for the earlier Kev-8B against Jev’s 0.857. The current Kev-9B closes most of that distance at 0.822, narrowing the gap from 0.061 to 0.035. Kev never saw those sources during training, and nobody outside TypeSafe knows whether Jev did. Jev publishes no test-column figure at all, so every number here is development-set only.

Kev-9B already wins some external work. On a 900-ticket support set, the authors report routing accuracy of 0.952 against Jev’s 0.897. Jev still leads on SemIf’s 144 authored decisions, 0.965 against 0.917.

Accuracy is not where Jev holds its lead. On the same new-source suite, Jev’s Brier score is 0.211 against Kev-9B’s 0.286. A Brier score rewards confident answers that turn out right and punishes confident answers that turn out wrong, so lower is better.

That is the second open-weights comparison to land the same way. Laya’s specialist beat Jev on accuracy, 0.766 against 0.727, and still reported the worse calibration error, 0.213 against 0.144. In both comparisons on record, Jev keeps the calibration edge, including the one where it loses the accuracy column. The only model here that beat Jev on both counts is Haiku, which is neither open nor cheap.

My read: Jev has the stronger case for working well out of the box and the better-behaved confidence number, while open weights offer a path to a task-specific lead once you fine-tune them on your own data. For anything you intend to threshold, that split is the part that matters. Accuracy tells you whether to use a model. Calibration tells you whether you can set a threshold on it at all.

The interface itself is already portable. Kev serves a TypeSafe-compatible endpoint, so the same client can point at a local server without rewriting the questions. The openJev-verdict-2.0 author reports fine-tuning an approximately 150M-parameter model in 8.8 hours on a consumer laptop GPU. That is self-reported, but it shows how cheap specializing has become.

The safety tasks themselves are not new. WildGuard, an open 7B model published in 2024, already covers prompt harm, response harm, and refusal detection.

I would not look to the request format for a moat now that independent implementations have copied it within the launch week. This deserves its own post, and I will come back to it.

What I Am Actually Excited About

Jev accepts state and bounded questions, then returns typed answers with probabilities.

A bounded semantic question is one where you fix the possible answers in advance, so the model picks among them instead of writing prose.

That is useful when your application needs a judgment it can consume directly.

The schema guarantees the answer’s form, not its correctness. But it gives me another implementation to test before asking a general-purpose model to produce a full assessment.

In Verification Is a Budget, Not a Default, I argued for choosing the verification approach that meets the required bar rather than automatically buying another large-model call.

Jev is a new candidate for the cheaper rungs of that ladder.

There Are Two Costs Here

The Dollars

TypeSafe lists $0.042 per million input tokens, with no output charge. That is vendor-reported and unreproduced here.

A community phishing benchmark provides a comparison using recorded token usage and list prices:

Source: Jev phishing benchmark. Community-reported usage priced at list rates. Haiku is claude-haiku-4-5. Jev’s request included multiple questions; Haiku’s original request asked for the broad verdict.

That is about 12 times less money spent on the first check.

The budget can reach more cases, or support more checks on each case. It can also leave more room for expensive review where the first check needs help.

The Waiting

The same benchmark reports a separate advantage in elapsed time:

Source: Jev phishing benchmark. Author-reported measurements from one machine in France, using sequential calls over a reused connection. These include network time.

p50 is the typical call. p95 shows the slow end that users will still experience regularly.

Speed matters separately from the token bill. If a check fits your latency budget before an action executes, you can use it to affect that action. A check applied only to sampled, completed traces cannot prevent those completed actions.

Openlayer’s jevals README adds a measured workflow comparison:

Evaluation implementation	Cost per 1,000 samples	Wall time for the 20-sample run
Ragas with gpt-4.1-mini	$2.60	22 to 35 seconds
jevals with gpt-4.1-mini emulating the decision interface	$0.46	4 seconds
jevals with Jev	$0.03	0.8 seconds

Source: jevals README, Numbers. Vendor-run and unreproduced. Measured on September 20, 2026, using 20 rows from a small RAG dataset shipped with the package, by a company that sells evaluation tooling. Costs use list pricing.

The README reports agreement on the verdicts across the measured implementations. Even keeping gpt-4.1-mini, changing the evaluation implementation reduced the reported cost and runtime. Jev reduced them further.

That is a reason to inspect how we build the evaluation, as well as which model answers it.

It also connects to September’s coverage argument. Lower cost can make inspecting more activity practical. Lower latency can put more of that inspection before execution.

Two honest caveats before I turn those wins into an architecture.

First, these are narrow comparisons: synthetic emails and a vendor’s own 20-row RAG sample measured on one day. RAG means the system retrieves documents before answering, so that sample tests a different job than phishing. The jevals comparison changes the evaluation implementation, and its README records a completion-count mismatch in Ragas’ answer-relevancy call. Treat it as a comparison of the tested implementations, rather than an isolated measure of model superiority.

Second, the budget belongs to the complete path. Deferred cases still need processing. Errors still need correcting. Measure those costs alongside the first-call wins.

Then I Read the Accuracy Results

This is where I stopped believing the replacement story.

On the same synthetic-email benchmark, Jev’s broad verdict achieved 62.6% accuracy against Haiku’s 81.3%. Jev’s expected calibration error was 0.154 against Haiku’s 0.097, where lower is better. Haiku was both more accurate and better calibrated on this task.

Calibration asks whether the confidence number deserves to be believed. A model that says it is 90% confident should be right roughly nine times out of ten on cases like that.

Source: Jev phishing benchmark. The first three bars are broad-verdict accuracy on the full dataset. The last three are held-out results from the separately fitted classifier experiment, which is not the same evaluation setup.

I cannot use that benchmark to celebrate the cost and latency, then omit what it says about the decision.

The idea is to let the model handle only the cases it is most sure about, and send everything else to review.

Selecting only higher-confidence predictions did not automatically rescue Jev.

At a predicted-class probability threshold of 0.90, Jev retained 30.8% of cases at 73.9% accuracy. Haiku retained 55.4% at 82.5% accuracy. Haiku handled more cases and made proportionally fewer errors within its retained subset. Jev’s confidence bins were also non-monotonic.

Non-monotonic means more confidence did not reliably mean more accuracy, so the number could not reliably sort the easy cases from the hard ones.

Those thresholds use the probability assigned to the predicted class, not Jev’s separate confidence field.

Keep three numbers apart: the model’s confidence score, the real chance the answer is right, and the threshold your application picks. A high confidence score is not permission to act.

The more interesting result comes from changing the check.

The hosting-list rule from the opening achieved 91.6% accuracy, beating both Jev’s 62.6% and Haiku’s 81.3% broad verdicts. The rule required no model and no fitting to labels.

In Verification Is a Budget, Not a Default, I wrote that you should bench small models against a deterministic baseline before assuming they beat a regular expression. Here was a source giving me a reason to follow my own advice.

The author then asked five narrow signal questions and combined the answers with logistic regression, fitting on one half of the dataset and evaluating on the other.

Think of the logistic layer as a small statistical combiner that learns how much weight to give each signal. Held-out means the second half was not used to fit that combiner, so it is the closer test of whether the pattern survives beyond the examples used to build it.

A false positive here means a legitimate email gets flagged as phishing.

Inputs to the classifier	Held-out accuracy	False-positive rate
Jev’s five atomic signal probabilities	95.0%	7.0%
Haiku’s answers to the same five questions	93.2%	5.4%
Two non-AI heuristic features	91.8%	0.2%

Source: Phishing benchmark, Control 2. Classifiers fitted on half A and evaluated on half B. The Jev-versus-Haiku paired comparison returned p = 0.0630, not significant at the conventional 0.05 threshold.

A p-value is roughly the chance of seeing a gap at least this large if the two approaches were really equal. At 0.063, the gap between Jev and Haiku here is not strong enough to call either one better.

The broad-verdict results describe the full dataset. The fitted-classifier results describe its held-out half. This is not a matched claim that decomposition alone caused an improvement from 62.6% to 95.0%. The author also designed the features with knowledge of the synthetic dataset’s URL patterns.

There is a useful design to investigate:

The logistic layer is another predictive model. Writing it in Python does not turn it into an authorization rule.

What changes my thinking is that the right question may be less “Which judge should replace this judge?” and more “Why did I ask one broad question in the first place?”

Start With the Cheapest Check That Clears the Bar

Selective automation asks: Which cases can clear this check without another round of verification?

Start with the least expensive approach that demonstrates acceptable performance for the decision. Escalate the cases outside its qualified scope. Qualified scope means the cases you have actually shown it handles well enough, not the cases you hope it handles.

Illustrative design. Required approvals remain required on every path.

For a check that needs substantial reasoning, the LLM judge may remain the first qualified option. Do not insert a cheaper model merely to say you used one.

But for a bounded check, the large judge should earn its position rather than inherit it.

This makes July’s verification ladder operational at the level of individual requests. Two cases in the same workflow can need different amounts of additional verification.

Take a refund request. One customer clearly identifies an order. Another gives conflicting instructions about several purchases. You might qualify a simpler interpretation check for the first subset while retaining additional review for the second.

Both paths retain eligibility checks and authorization.

A useful confidence ranking can support this selection without every numerical value being a literal probability of correctness. Selective classification already studies that trade-off: letting a model answer only the cases it handles well and routing the rest somewhere else. What matters operationally is the measured error among the cases your rule accepts, alongside how much work it accepts.

Calibrating down lowers the cost of meeting the bar, not the bar itself.

This Is Bigger Than Evals

The interface becomes relevant wherever a system needs to interpret uncertain information before applying a rule.

For customer-facing conversational systems, input and output safety checks belong in the baseline design. When those checks cost near zero and run in parallel, there is less reason to ration them.

For fraud detection, I would investigate narrow semantic signals. Does a customer’s message describe pressure to transfer money? Does a payment-change request conflict with earlier instructions?

Those are hypotheses to evaluate alongside transaction evidence. Payment authorization remains a separate decision.

The same pattern could support document triage or route a disputed-payment request to the right process. An entitlement pre-check might identify which permission needs checking. The authoritative entitlement lookup must still decide whether access exists.

I have not tested these. They are ideas worth evaluating, not results.

As I wrote in the September post: “Don’t turn a known transaction limit into an LLM prompt.”

Keep the known limit in code. Test the model on the interpretation that code cannot supply.

The local implementations widen that investigation. jevals supports Kev and Laya backends, allowing evaluation definitions to run against local models rather than only a hosted Jev API.

For regulated industries, that creates a different self-hosting and data-residency option to assess. Qualify the local model’s behavior and the surrounding controls separately.

A shared request format makes implementations easier to exchange. Each backend still needs its own threshold validation. The jevals documentation explicitly calls for recalibration when changing backends.

What the Confidence Number Does Not Tell You

I am excited about Jev, so here is what should temper that.

First, a quick primer on Jev’s three answer types. Choice picks one option from a list. Score rates something on an ordered scale. Noul returns the probability that a yes/no statement is true.

PrimeLine reports calibration errors of 0.012 for Noul, 0.086 for Choice, and 0.254 for Score. The populations and confidence constructions differ. Its separate applied-task corpora came from one developer’s project; 56.5% of the 4,000-commit source history carried a Claude co-author trailer. The site discloses AI-assisted writing. That contamination caveat concerns the applied corpora, not the public calibration datasets.

Haiku beat Jev on one applied job in the same report.

The useful repair was narrower: one yes/no question before a broad classification recovered 23 of 53 known failures, or 43.4%. No new errors appeared on a length-matched control, which was not matched on category or confusability. That is a targeted failure repair.

Then there is the arithmetic behind confidence.

Stanislav Yurin’s analysis gives Choice confidence as:

C = (N * p_max - 1) / (N - 1)

N     = number of answer options
p_max = probability of the leading answer

At a fixed top probability of 0.60, confidence is 0.200 with two options and 0.579 with twenty. The answer schema changed while the top probability stayed fixed.

Option-count qualification rule: A threshold belongs to the exact question and answer schema. Changing the option count requires revalidation.

For Score, Yurin explains a different calculation based on distance from the most probable rubric level. His September 20 update links TypeSafe’s published implementation confirming both formulas. Keep their meanings separate when setting thresholds.

Repeatability is where Jev earns some of my excitement.

LangChain’s vendor-reported, unreproduced experiment found Jev’s mean per-case quality-score variance 92 to 913 times lower than three LLM judges. That was five fixed cases, each scored 100 times. Lower variance means that when you send the same case again, the score moves less.

Alongside that, Yurin reports ten identical calls moving confidence between 0.84 and 0.88 on an ambiguous item. On a closer case, the selected label changed.

Repeated scoring can be far more stable while close calls still flip. Your threshold determines whether the remaining variation changes the action your system takes.

Prompt injection belongs in the test set too. That is when the content being reviewed carries text designed to hijack the instructions the model is following. PrimeLine’s 40 matched pairs produced 22.5% misclassification on injected versions, with mean absolute score movement of 0.193. One innocent message moved from 0.04 to 0.66 on suspiciousness after injection: a false alarm, not an action bypass. TypeSafe documents the adversarial-input risk.

The Default That Becomes Your Policy

Qualification asks: What evidence supports this exact check controlling this exact action?

The unit is not “Jev approved.” It is the model version together with the question, the supplied state, and the selection rule. The permitted action and failure behavior belong in the same record.

In the September framework, I already required “defined behaviour when the evaluator is unavailable.”

jevals already hit this case in its own testing.

During Openlayer’s vendor-run, unreproduced testing, the gateway hung on some connections. The client retried with backoff, waiting longer between each attempt, and p95 latency for that run reached one minute. The README records the slow path during its September 20 testing.

Its documented gate behavior retries backend failures, then lets the call through by default if the backend remains unavailable. The README explicitly recommends on_error="block" for gates in front of anything irreversible.

That default is fail-open. Fail-open means the protected action proceeds when the evaluator is unavailable. Fail-closed means the action is blocked instead.

Source: jevals gate documentation.

I can understand an availability-friendly default in a development library.

But the recommendation is block, and the default is allow. Following the recommendation requires configuration. Put that gate in front of a tool that moves money, and the configuration becomes a governance decision.

Blocking also needs a recovery path and a defined way to handle unresolved work.

This is why I would put qualification alongside coverage, binding, and closure:

Qualification applies across the control, rather than adding another sequential runtime step.

Your platform can enforce the wrong judgment exactly as configured. Lower inference cost increases the importance of qualifying the judgment before expanding its reach.

The June distinction remains: a score is not a decision.

The jevals README includes a concrete example. A $49 refund that the customer explicitly requested still escalates, on the destructive signal alone. The example’s policy sends money-moving actions to a human regardless of grounding. That is Openlayer’s reported gate behavior, not an independently reproduced control test.

More confidence about the customer’s request would not remove the approval requirement. The judgment answers one question. The policy governs what can happen next.

Cheap typed judgment makes it more practical to place scores throughout an application. That raises the stakes on the decision layer.

What To Do

1. Pick one bounded check and define the acceptance criteria first. Specify the errors you cannot tolerate and the cases outside scope. Keep mandatory approvals separate from any optional interpretation review.

2. Benchmark against code and the existing judge. Include the simplest credible baseline. Run Jev in shadow mode, meaning it scores real traffic but its answers change nothing, so you can compare safely. Compare the work each approach can handle at the same error requirement.

3. Select on development data, then freeze the configuration. Test the frozen question, schema, and threshold on held-out cases. Report error within the automatically handled subset and the amount of work retained. Do not use the same cases to keep adjusting the threshold and declare success.

4. Measure the complete execution path. Account for deferred cases and retries. Test the fallback on the cases it actually receives. Measure dollars separately from latency, including the slow end of the distribution.

5. Test the ways the gate can fail. Include prompt injection and repeated calls near the threshold. Make backend-outage behavior explicit. Requalify when changing the model or answer options.

6. Keep checking the cases that pass automatically. Do not review only escalations. Preserve a route back to the previous review process, and identify who can authorize a threshold change. Expand automation only after the accepted cases continue to meet the criteria you set.