System One benchmark

An independent benchmark of TypeSafe's Jev against every open System One alternative: decision accuracy, calibration (ECE) and latency on two suites, 2,123 judgments, same harness for every row.

Every open-weights alternative to TypeSafe's Jev, ranked by measured decision accuracy on the 2,000-decision typed-decisions suite: LAYA, Von, GLiNER2, GLiClass, OpenJEV, plus the Jev-shaped checkpoints not yet run.

The benchmark for System One models: decision models that answer typed questions with calibrated probabilities. Jev and every open alternative on the same 2,000-decision suite, with accuracy, wins over the majority baseline, ECE and latency.

One table, every System One model: Jev, LAYA (three checkpoints), Von, OpenJEV, seven GLiNER2 and seven GLiClass checkpoints. Decision accuracy, calibration and latency on the Metatext suite, extraction F1 where the model has a span head.

The open reproductions of Jev (OpenJEV, LAYA, Von, and the Kotoba, Decider, Kev, Gavel checkpoints) explained by mechanism and measured on the same suite as the real thing.

Decision models with open weights and a permissive license, ranked by measured accuracy with their latency on four CPU cores, for teams that cannot send their state to a hosted API.

The best open-weights System One model by measured decision accuracy on the typed-decisions suite, with calibration and latency, and the surveyed checkpoints still to run.

Expected calibration error of Jev and every open System One model, raw and after one fitted temperature per primitive judged on a held-out fold. TypeSafe publishes no calibration number; this page does.

System One models ranked by expected calibration error on the Metatext suite, untuned, with the held-out ECE after temperature scaling where it was fitted.

System One models ranked by p50 latency per three-question case on a four-core CPU, against Jev's hosted round trip.

Micro-F1 on an eight-case entity extraction task for the models that have a span head. Jev, LAYA, Von and GLiClass never return spans; only the GLiNER family does.

Leaderboard for LAYA's public typed-decisions suite: Jev, LAYA, Von, GLiNER2 and GLiClass on 2,000 decisions across four workflows, with per-question detail and calibration.

Leaderboard for the Metatext suite: three hand-written workflows, seven questions, 123 judgments, plus extraction, for Jev and every GLiNER2, GLiClass, LAYA, Von and OpenJEV checkpoint.

The original System One model: a hosted decision model that takes a state and typed questions (Choice, Score, Noul) and returns a calibrated probability distribution per question in one round trip, never generating text.

The open-weights System One: takes the Jev request verbatim and returns the Jev answer. One forward pass scores every option of every question; ships with a fitted temperature per primitive.

The English LAYA checkpoint fine-tuned on the typed-decisions suite. In-domain on that suite by construction; the number to read is how far a fine-tune closes the gap to Jev.

The 100+ language LAYA checkpoint on an mmBERT-base backbone. Ships with temperature [1, 1, 1], so it runs uncalibrated out of the box.

Jev's API over an NLI classifier: every option becomes a hypothesis and the distribution is a softmax over entailment logits. Its config matches tasksource/ModernBERT-large-nli entry for entry.

The open-source implementation of Jev's mechanism: score each allowed answer by its log-probability under a chat LLM, then assemble the distribution. Measured here on a 0.5B backbone, so it is a scale substitution, not a like-for-like row.

The production GLiNER2 checkpoint: entities, classification tasks and structured records in one pass. The best open model on the Metatext suite and the only family that also extracts spans.

The smaller v1-generation GLiNER2 checkpoint.

The multilingual v1-generation GLiNER2 checkpoint.