Confidence intervals and resampling

Model Auditor exposes inference through InferenceConfig. Intervals are pointwise and condition on supplied predictions and thresholds; they do not include model training, threshold-selection, or multiplicity uncertainty.

Automatic interval selection

With the default method="auto" and IID sampling:

• binomial rates use Wilson intervals;
• rate_interval="exact" selects Clopper–Pearson intervals instead; and
• confusion-membership odds ratios use conditional exact intervals.

Other eligible metrics use percentile resampling. method="bootstrap" requests resampling for eligible metrics that would otherwise use analytic intervals.

The LevelMetric.score always remains the metric calculated on the original observed rows. Count metrics and declared-but-unobserved categorical placeholders do not receive intervals.

Configuration

from model_auditor import InferenceConfig

inference = InferenceConfig(
    confidence_level=0.95,
    method="auto",
    rate_interval="wilson",
    resampling="cluster",
    cluster="patient_id",
    random_state=42,
    missing="exclude",
    min_valid_fraction=0.95,
    min_resamples=100,
)

• iid resamples rows;
• stratified conditions on observed truth-class counts; and
• cluster resamples whole cluster IDs while retaining the row-weighted estimand.

random_state creates a local NumPy generator. Evaluation does not consume or modify NumPy’s global random state.

Diagnostics

LevelMetric records the interval method and status, requested and valid resample counts, nonfinite resamples, denominator, and estimate status. An interval is withheld when too few finite replicates survive min_valid_fraction and min_resamples.

Use to_numeric_dataframe() to export those fields with support and provenance. Undefined metrics remain NaN rather than acquiring an interval from a subset of apparently successful replicates.

Choosing a resample count

Use None to disable all intervals:

results = auditor.evaluate_metrics(
    score_name="risk_score",
    n_bootstraps=None,
)

Use a positive integer to request intervals:

results = auditor.evaluate_metrics(
    score_name="risk_score",
    n_bootstraps=2000,
    inference=InferenceConfig(random_state=42),
)

Zero is not a supported shorthand for no intervals. More iterations reduce Monte Carlo noise but do not fix bias, sparse support, dependence, or poor sampling design.

Scope

Cluster resampling changes the sampling unit, not the row-weighted estimand. Stratification conditions on observed class counts. Neither design refits the model, adjusts for data-driven threshold or subgroup selection, supplies simultaneous coverage, or turns descriptive subgroup differences into causal or fairness conclusions.