Calibrated AI Workflows
Three different quantities
The winning Choice probability says how much probability the model assigns to an option. TypeSafe's confidence summarizes concentration relative to a uniform distribution. With three options and a winner at 0.9, confidence is 0.85. Neither is an observed accuracy score.
Calibration is a property measured across examples: among answers assigned probability near 0.8, is the correct outcome observed about 80% of the time? A confidently wrong classifier can have valid output and low transport error while failing this test. Evaluate the winning probability against independent correctness labels, not the normalized confidence statistic.
Choice confidence is (p_max − 1/n) / (1 − 1/n). Score confidence uses distance from the modal level relative to a uniform reference, and Score itself is an expected level index. Noul returns only its yes probability. You may derive |2p − 1| as a concentration measure, but it is not a returned field or a replacement for calibration.
Our 100-ticket cohort deliberately separates clear cases, ambiguous cases and human requests. The labels and probabilities are invented. That lets you observe a known failure without implying Jev actually has the accuracy you see here.
Choice C = (max(p) - 1/n) / (1 - 1/n) Score E = sum(i * p_i) Score C = max(0, 1 - sum(p_i * |i - mode|) / MAD_uniform) MAD_uniform = mean(|i - (n-1)/2|) Noul p = probability of yes; optional concentration = |2p - 1| Calibration: compare p with outcomes in a group. Policy: choose actions using evidence and the cost of error.
Sources and review dates
- TypeSafe confidence formulas · reviewed 2026-10-11
- Confidence routing · reviewed 2026-10-11
- Composite judgments in code · reviewed 2026-10-11
- RAG passage classification · reviewed 2026-10-11
- Guo et al.: On Calibration of Modern Neural Networks · reviewed 2026-10-11
- Geifman and El-Yaniv: Selective Classification · reviewed 2026-10-11
- Independent classification study (September 29 preprint) · reviewed 2026-10-11
- Independent annotation study (October 6 revision, preprint) · reviewed 2026-10-11
- TypeSafe's own evaluations · reviewed 2026-10-11
- Model versions, pricing and limits · reviewed 2026-10-11
8 components9 connections0:00
Recording…