Evaluate calibration and decisions

Calibration and decision evaluation require a registered score containing finite probabilities in [0, 1] and a registered binary outcome. They do not require a stored classification threshold.

Evaluate calibration

from model_auditor import InferenceConfig

calibration = auditor.evaluate_calibration(
    "risk_probability",
    bins=10,
    n_bootstraps=2000,
    inference=InferenceConfig(random_state=42),
    cohort="held-out",
)

calibration.bins contains every fixed equal-width bin for each feature level, including empty bins. It reports bin edges, support, mean prediction, observed frequency, pointwise frequency bounds, and interval diagnostics.

calibration.summary is a ScoreEvaluation containing Brier score, binary log loss, calibration intercept, and calibration slope. Intercept and slope require both truth classes, interior probabilities, and an identified logistic fit. Boundary probabilities, constant predictions, separation, or single-class levels can therefore return NaN.

Evaluate net benefit

curve = auditor.decision_curve(
    "risk_probability",
    thresholds=[0.05, 0.10, 0.20, 0.30],
)

For each threshold probability t, the result reports:

net benefit = TP / N - FP / N × t / (1 - t)

It also includes act-all and act-none comparators, selection rate, and sample size. Thresholds must be a nonempty finite sequence strictly inside (0, 1). The curve is descriptive only: it provides no confidence bands or automatic threshold selection, and t must encode a meaningful false-positive tradeoff for the target decision.