Compare models and groups

Model Auditor can report metric differences or ratios while preserving paired rows and subgroup overlap inside resamples.

Compare two scores

from model_auditor import InferenceConfig
from model_auditor.metrics import AUROC, Sensitivity, Specificity

auditor.add_score("candidate", threshold=0.45)
auditor.add_score("reference", threshold=0.50)
auditor.set_metrics([AUROC(), Sensitivity(), Specificity()])

comparison = auditor.compare_scores(
    "candidate",
    "reference",
    contrast="difference",
    n_bootstraps=2000,
    inference=InferenceConfig(random_state=42),
    cohort="held-out",
)

For difference, each point is candidate minus reference. For ratio, it is candidate divided by reference; a zero or undefined reference produces NaN. Thresholds can be overridden independently with threshold= and reference_threshold=. Both models are evaluated on the same rows, and metric inputs are recomputed inside every shared resample.

Compare feature levels

group_comparison = auditor.compare_groups(
    "candidate",
    feature="region",
    reference="North",
    contrast="difference",
    n_bootstraps=2000,
    inference=InferenceConfig(random_state=42),
)

Every observed non-reference level is compared with the named reference. With FPR or FNR, these are class-conditional error-rate contrasts, unlike the confusion-membership enrichment odds ratios returned by evaluate_errors().

Both methods return a ScoreEvaluation. Intervals are pointwise and are not adjusted for multiple comparisons, model selection, threshold tuning, or subgroup discovery. Undefined component metrics produce undefined contrasts.