Best Calibrated Decision Models
A well-calibrated decision model is right about 80% of the time when it says 80%, so its probabilities can be used as thresholds directly. This ranking sorts models at or above the median Decision Index score by raw expected calibration error (lower is better), with accuracy as the tie-break, so a model cannot rank by being uniformly unsure.
Ranked list (top 25)
| # | Model | Decision Index | ECE | p50 latency | Per 1k decisions | Params |
|---|---|---|---|---|---|---|
| 1 | Nace.AI Drex v2.0 (35B-A3B) Nace.AIOpen weights | 55.9 | 0.018 | 276 ms | $0.012 | 35B |
| 2 | Decider chat · Qwen3.6-27B MapikaOpen weights | 51.3 | 0.021 | 84 ms | — | 28B |
| 3 | Kev 27B Jared PalmerOpen weights | 58.8 | 0.022 | 135 ms | — | 28B |
| 4 | Decider 35B-A3B MapikaOpen weights | 48.5 | 0.023 | 101 ms | — | 36B |
| 5 | Torchcast Decision 27B Torchcast AIOpen weights | 59.9 | 0.024 | 100 ms | — | 28B |
| 6 | Xor 1.2 Juspay TechnologiesOpen weights | 51.1 | 0.024 | 150 ms | — | 36B |
| 7 | Bespoke Nimble 9B v2 Bespoke LabsOpen weights | 41.0 | 0.024 | 77 ms | — | 10B |
| 8 | Decider 31B (NVFP4) MapikaOpen weights | 64.6 | 0.024 | 72 ms | — | 33B |
| 9 | Cloudflare clef-flash CloudflareOpen weights | 47.6 | 0.025 | 53 ms | $0.0042 | 10B |
| 10 | OneJev1.5-9B OmniJevOpen weights | 53.3 | 0.029 | 111 ms | — | 9B |
| 11 | vLLM-SR Decision 2.0 Nox 4B vLLM Semantic RouterOpen weights | 45.0 | 0.030 | 34 ms | — | 4B |
| 12 | Sieve 4B Sthānika AIOpen weights | 46.1 | 0.031 | 93 ms | — | 5B |
| 13 | PurpleMIST-Flash-1.0 Olaverse LabOpen weights | 46.8 | 0.032 | 53 ms | — | 8B |
| 14 | Cloudflare clef CloudflareOpen weights | 53.1 | 0.040 | 102 ms | $0.0084 | 28B |
| 15 | Fastino GLiDE Fastino | 60.2 | 0.040 | 98 ms | $0.030 | 28B |
| 16 | Cloudflare clef-omni (30B-A3B) CloudflareOpen weights | 48.3 | 0.040 | 191 ms | $0.030 | 32B |
| 17 | Blink v0.3 26B-A4B (NVFP4) Pixilab.aiOpen weights | 57.8 | 0.040 | 25 ms | — | 26B |
| 18 | vLLM-SR d3 (27B) vLLM Semantic RouterOpen weights | 66.3 | 0.043 | 97 ms | — | 26B |
| 19 | Wald-26B-A4B ORG2 AIOpen weights | 58.3 | 0.043 | 90 ms | — | 26B |
| 20 | Sieve 9B Sthānika AIOpen weights | 44.6 | 0.043 | 50 ms | — | 10B |
| 21 | reflex Qwen3.8-27B-FP8 (pmscr) kshetrajna12Open weights | 54.7 | 0.045 | 149 ms | — | 28B |
| 22 | Kev 9B v2 Jared PalmerOpen weights | 43.3 | 0.045 | 60 ms | — | 10B |
| 23 | Darwin-27B-ZTC FINAL_BenchOpen weights | 62.5 | 0.045 | 104 ms | — | 26B |
| 24 | FlyMy.AI Decision 4B v1.6 FlyMy.AIOpen weights | 43.6 | 0.046 | 27 ms | — | 5B |
| 25 | Decider chat · Gemma-4-31B MapikaOpen weights | 57.6 | 0.047 | 109 ms | — | 33B |
Scores, calibration (ECE) and latency from the Jev Decision Index; cost per 1,000 decisions at 200 input tokens each. Updated daily.
More decision-model rankings
Frequently asked questions
Which decision model is best calibrated?
Nace.AI Drex v2.0 (35B-A3B) has the lowest expected calibration error among models at or above the median Decision Index score (ECE 0.018).
What is expected calibration error (ECE)?
ECE groups predictions by stated confidence and measures the average gap between confidence and accuracy in each group. 0 is perfect; below about 0.05 is good enough to use probabilities as thresholds.
Can calibration be fixed after the fact?
Often, yes: fitting one temperature per question type on a small labelled set usually lowers ECE a lot. The ranking here uses raw ECE as published by the Decision Index.