Evaluate calibration and decisions

Calibration and decision evaluation require a registered score containing finite probabilities in [0, 1] and a registered binary outcome. They do not require a stored classification threshold.

Evaluate calibration

from model_auditor import InferenceConfig

calibration = auditor.evaluate_calibration(
    "risk_probability",
    bins=10,
    n_bootstraps=2000,
    inference=InferenceConfig(random_state=42),
    cohort="held-out",
)

calibration.bins contains every fixed equal-width bin for each observed feature level, including empty bins. Bins are left-closed/right-open except the final bin includes 1. Declared unobserved categories appear only in the summary. The bins table reports edges, support, mean prediction, observed frequency, pointwise frequency bounds, and interval diagnostics.

calibration.summary is a ScoreEvaluation containing Brier score, binary log loss, calibration intercept, and calibration slope. Intercept and slope require both truth classes, interior probabilities, and an identified logistic fit. Both fits return NaN for boundary probabilities, single-class levels, or failed optimization. Constant predictions and separation additionally prevent the slope fit; the offset-only intercept can remain estimable with constant interior probabilities.

Evaluate net benefit

curve = auditor.decision_curve(
    "risk_probability",
    thresholds=[0.05, 0.10, 0.20, 0.30],
)

For each threshold probability t, the result reports:

net benefit = TP / N - FP / N × t / (1 - t)

It also includes act-all and act-none comparators, selection rate, and sample size. Thresholds must be a nonempty finite sequence strictly inside (0, 1). The curve is descriptive only: it provides no confidence bands or automatic threshold selection, and t must encode a meaningful false-positive tradeoff for the target decision.