Skip to content
Learn/

Calibrated AI Workflows

1 / 9

Three different quantities

The winning Choice probability says how much probability the model assigns to an option. TypeSafe's confidence summarizes concentration relative to a uniform distribution. With three options and a winner at 0.9, confidence is 0.85. Neither is an observed accuracy score.

Calibration is a property measured across examples: among answers assigned probability near 0.8, is the correct outcome observed about 80% of the time? A confidently wrong classifier can have valid output and low transport error while failing this test. Evaluate the winning probability against independent correctness labels, not the normalized confidence statistic.

Choice confidence is (p_max − 1/n) / (1 − 1/n). Score confidence uses distance from the modal level relative to a uniform reference, and Score itself is an expected level index. Noul returns only its yes probability. You may derive |2p − 1| as a concentration measure, but it is not a returned field or a replacement for calibration.

Our 100-ticket cohort deliberately separates clear cases, ambiguous cases and human requests. The labels and probabilities are invented. That lets you observe a known failure without implying Jev actually has the accuracy you see here.

Choice C = (max(p) - 1/n) / (1 - 1/n)
Score E = sum(i * p_i)
Score C = max(0, 1 - sum(p_i * |i - mode|) / MAD_uniform)
MAD_uniform = mean(|i - (n-1)/2|)
Noul p = probability of yes; optional concentration = |2p - 1|

Calibration: compare p with outcomes in a group.
Policy: choose actions using evidence and the cost of error.
Sources and review dates

8 components9 connections0:00

Traffic20req/s
Traffic
20req/s
p50
165ms
p99
406ms
Errors
0.09%
Dropped
0.0req/s
Cost
$3,034/mo