Where a calibrated classifier fits in PolicyCheck

The headline

Recommendation

Run a one-week offline benchmark on the seed corpus, then two weeks in shadow mode on document classification and endorsement matching. Cut over behind flags only once calibration is proven and TypeSafe has answered five compliance questions. No tenant data leaves the platform until then. Keep Amazon Bedrock as the permanent fallback.

What Jev is

Jev is the first "System One" model from TypeSafe AI, released 15 September 2026. It is a decision model, not a language model told to output JSON. You give it text plus a fixed set of questions whose answers you define in advance. It returns every answer in a single parallel pass, each with a calibrated probability distribution.

It cannot write a sentence, and that is the point. It cannot invent a label that is not in your list, and its confidence numbers are trained to mean something: answers given 80% should be right about 80% of the time.

Three question types

Many questions of any type can share one input in one request. Adding a question adds input tokens but almost no latency, and each answer is independent of the others.

What a request looks like

// one request, three independent answers
const r = await client.systemOne({
  state: { filename: "END--BAA-2025-Territory.pdf", text: first16kChars },
  questions: {
    docType: choice("What kind of Lloyd's market document is this?", {
      BAA: "A binding authority agreement between managing agent and coverholder",
      BINDER_ENDORSEMENT: "An endorsement that amends a binding authority agreement",
      POLICY_ENDORSEMENT: "An endorsement that amends an individual policy",
      OTHER: "None of the above",
    }),
    isReinsurance: noul("The document concerns reinsurance"),
    scanQuality: score("How readable is the extracted text?",
      ["Clean digital text", "Minor OCR noise", "Heavily garbled"]),
  },
});

r.answers.docType.choice;      // "BINDER_ENDORSEMENT"
r.answers.docType.confidence;  // 0.91
r.answers.isReinsurance.noul;  // 0.03

TypeScript and Python SDKs exist, plus a Pydantic AI adapter. Input is text only; scanned content must be transcribed first.

Published numbers

Vendor-supplied. We have not reproduced these independently.

What it cannot do

• Generate anythingNo summaries, free-text extraction or explanations. The pattern is: find candidates with regex or an LLM, then let Jev pick.
• Read between the linesNegations are taken literally. A question whose "yes" means a bad outcome underperforms.
• Count or calculateArithmetic and counting are unreliable and worsen with list size.
• Compare datesPick date values with Choice; do the ordering in code.
• Ignore noiseAccuracy drops as irrelevant text is added. Filter first.
• Act as a security boundaryUser-controlled input can steer it.
• Match English elsewhereOther languages are handled, but not equally well.
• Handle huge label setsMore than 255 options needs a two-stage scheme.

How PolicyCheck classifies today

There is exactly one LLM classifier on the per-document hot path: document type plus product type, via detectMetadata in llm-service. Everything else is deterministic (regex, keyword tables, alias maps, database lookups) or an LLM call that only runs when a cheaper step fails. All LLM traffic runs through llm-service and LiteLLM to Amazon Bedrock, with nova-micro as the default classification model.

Where classification sits in ingestion

1. Slice sync and exact-checksum duplicate check
2. Text extraction (Docling, MinerU, pdf-parse, OCR) alongside PII and quality pre-checks
3. Classification: LLM document and product type, then a stack of deterministic overrides main opportunity
4. Layout normalisation, canonical markdown, parse-completeness gate
5. Template verification: LMA template classifier and section tagging
6. Near-duplicate check, language detection
7. Metadata: insurer, product and coverholder resolution opportunity
8. Extraction: LLM tiers, clause segmentation, term tagging
9. Hydration: class, currency, territory, occupancy normalisation; identity and date fallbacks
10. Endorsement overlay: operation classification, clause match opportunity
11. Asynchronous enrichment: number tagging, document agents, signature tags

Six findings from the survey

• Document classification confidence is not a measurement.It always returns 0.85 with no confirmation required. The early-classification threshold (0.7), the endorsement association gate (≥ 0.9) and the low-confidence tag therefore never change behaviour.
• One LLM call answers ten questions as free text, then five keyword layers second-guess it.Our own baseline on 503 labelled documents shows that without the LLM's answer, cover notes, both endorsement types and schedules of authority score 0%. The heuristics only work when the LLM has already done the job.
• Four LLM classification paths were designed and never connected.Product-type mapping, the endorsement fallback, chat agent routing and the chat shadow router. Each was most likely left off because an LLM call per document or per message was too slow or expensive. Jev's economics are the missing precondition.
• Insurer resolution creates duplicate records.When a fuzzy match fails, a new insurer is auto-created as pending review, so near-duplicates accumulate. A "same entity?" check over the top candidates is the cheapest fix.
• Vocabularies are duplicated.Class of business exists in three places and document type in five. We should settle one canonical list per concept before adding Jev, or it becomes a sixth.
• Inbound email classification is probably not reaching the LLM.The call targets the wrong route, omits a required field and reads a response field that does not exist. The filename fallback is doing all the work. Not verified at runtime, but three independent mismatches sit in the same call path.

Ranked opportunities

A deterministic rule gives the same answer every time, costs nothing and can be explained in an audit. On a Lloyd's-market platform that is worth keeping, so Jev is not a replacement for rules that work. We applied four rules, in order, to every touchpoint:

1. Replace
   Replace an existing LLM callThe decision is already model-made. Jev makes it calibrated, sub-second and unable to return an off-list label.
2. Fill slot
   Fill a slot designed for an LLM but never wiredFour such paths exist in the code today.
3. Leftovers
   Rule first, Jev only on what the rule gives up onA 0.3 passthrough, a default of "other" or "unknown", missing data, an auto-created record. The rule still runs first and still wins when it fires.
4. Keep
   Otherwise keep the ruleA heuristic that works is not replaced. Where we have no failure data, the item is marked Measure until we do.

Tier 1: do these first

Tier 2

Other LLM calls to replace

Rule first, Jev only on the leftovers

Where it does not belong

• Anything that writes text: extracted values, summaries, rationales, chat answers.
• Anything numeric: premium sums, date ordering, counting clauses.
• Safety, tenant isolation, authorisation, and final verdicts on rule-based authority checks.
• Non-English documents, until accuracy on them is measured.

Full touchpoint inventory

Every place the platform classifies or normalises, with its current approach and our recommendation under the four rules. "Hot" runs per document, page, email or chat turn. File references were checked against the codebase on 26 September 2026.

Pilot plan

Before any tenant document is sent

Until questions 1–3 are answered, the pilot uses only the Lloyd's seed corpus. No tenant data.

Phases

1. Phase 01 week

   Offline benchmark, no production change

   • Add a Jev runner to the existing baseline harness (503 labelled seed documents) for J1.
   • Report strict and family-level accuracy per document class, plus a calibration curve (stated confidence against observed accuracy).
   • Pass bar: match or beat the current system on every focus class, and beat the heuristics-only result by a wide margin on the four classes at 0%.
   • Model cost of the run: about $0.08.
2. Phase 12 weeks

   Shadow mode

   • Add Jev as a provider in llm-service, selectable per operation from the existing admin runtime settings.
   • Run J1 and J2 in shadow: log Jev's answer, confidence and latency alongside production.
   • Run J5 as a measurement only, beside the keyword chat router. No switch without disagreement data.
   • Measure agreement, disagreement categories, and p50 / p95 latency from eu-west-2.
3. Phase 2after sign-off

   Cut over J1, J2 and J6 behind flags

   Confidence-gated, with thresholds tuned from the Phase 0 calibration curve:

   ≥ 0.85Auto-accept 0.60 – 0.85Accept, ask for confirmation < 0.60Fall back to current path

   • Retire the two free-text translation chains once agreement is stable. The filename, frontmatter, reinsurance and body-text rules stay.
4. Phase 3as capacity allows

   Remaining opportunities

   • J3 and the rest of Tier 2. "Measure first" items only once we have failure data.

Risks and economics

Indicative model cost

Assumes ~4k tokens per document, 1.5k per chat turn and 2k per endorsement, at prices published 26 September 2026.

The current nova-micro classification is already inexpensive ($0.15 per million input tokens, $0.60 output), so direct model savings are modest. The value is in latency: sub-second answers let early classification stay on and skip full parses for supporting documents. And it is in real confidence, which activates the confidence-gated pipeline branches that exist in code but never fire.

Sources

• TypeSafe announcement
• Docs: primitives
• Docs: models
• Docs: System One
• MindStudio explainer
• Practical guide (DEV)
• Pydantic AI adapter
• Requesty explainer
• OpenRouter listing