Evaluate subgroup performance

This guide evaluates one binary-classification score overall and across multiple features, then verifies that the result contains both observed estimates and confidence intervals.

Prerequisites

• Model Auditor v0.1.16 is installed.
• Your DataFrame has one row per evaluation unit.
• The outcome is encoded as 0 and 1, or you have a complete mapping to those values.
• The score is continuous and oriented so larger values mean a more positive prediction.
• Missing values in the score, outcome, and audit features have been handled deliberately.

Prepare ordered feature levels

Use pandas categorical columns when report order matters or when declared-but-unobserved levels should remain visible.

import pandas as pd

data["age_group"] = pd.Categorical(
    data["age_group"],
    categories=["18–39", "40–59", "60–79", "80+"],
    ordered=True,
)

Model Auditor preserves the declared order. A category with no rows appears in metric tables with NaN scores and no confidence interval.

Configure the audit

Include support and class-balance metrics when you plan to interpret small subgroup estimates or annotate interval plots.

from model_auditor import Auditor, InferenceConfig
from model_auditor.metrics import (
    AUPRC,
    AUROC,
    FNR,
    FPR,
    MatthewsCorrelationCoefficient,
    Precision,
    Sensitivity,
    Specificity,
    nData,
    nNegative,
    nPositive,
)

auditor = Auditor()
auditor.add_data(data)

auditor.add_feature(name="age_group", label="Age group")
auditor.add_feature(name="region", label="Region")
auditor.add_feature(name="device_type", label="Device type")

auditor.add_score(
    name="risk_score",
    label="Risk score",
    threshold=0.5,
)

auditor.add_outcome(
    name="outcome",
    mapping={"positive": 1, "negative": 0},
)

auditor.set_metrics(
    [
        Sensitivity(),
        Specificity(),
        Precision(),
        AUROC(),
        AUPRC(),
        MatthewsCorrelationCoefficient(),
        FPR(),
        FNR(),
        nData(),
        nPositive(),
        nNegative(),
    ]
)

Validate mapped outcomes

add_outcome(mapping=...) uses pandas mapping. Any source value absent from the mapping becomes null; it is not rejected immediately. Check that the resulting source values are fully covered before evaluation.

Run the evaluation

A positive bootstrap count requests intervals for eligible metrics. Automatic IID inference uses Wilson intervals for binomial rates and diagnosed percentile resampling for other metrics.

results = auditor.evaluate_metrics(
    score_name="risk_score",
    n_bootstraps=1000,
    inference=InferenceConfig(random_state=20260831),
    cohort="held-out-2026",
)

The seed creates a local generator and does not modify NumPy’s global random state. The cohort identifier and inference policy are retained in result metadata.

Verify the result

Every evaluation contains an automatic overall feature and each feature you registered:

assert list(results.features) == [
    "overall",
    "age_group",
    "region",
    "device_type",
]

Inspect a single numeric estimate and interval:

metric = (
    results.features["region"]
    .levels["North"]
    .metrics["sensitivity"]
)

print(metric.score)
print(metric.interval)

A CI-eligible observed level should have a two-value interval. Count metrics and unobserved categorical placeholders retain interval=None.

Create a readable table

table = results.to_dataframe(
    n_decimals=3,
    metric_labels=True,
)

print(table)

This table formats values for display. Cells with intervals are strings such as 0.812 (0.744, 0.873). Use results.to_numeric_dataframe() for unrounded long-form estimates, support, diagnostics, and provenance.

Interpret subgroup results responsibly

A difference between subgroup point estimates is not automatically a statistically established or practically important disparity. Check:

• subgroup sample size and class balance;
• confidence-interval width and overlap;
• whether the feature was selected before or after seeing the result;
• whether threshold choice is appropriate for every subgroup;
• missing or unobserved feature levels; and
• external validity on a separate cohort.

Model Auditor calculates and organizes the evidence; it does not define an acceptability policy.

Common failures

Symptom	Likely cause	Resolution
“Please add data…”	add_data() was not called, or later state was replaced	Add the DataFrame before the outcome and rerun configuration
“Please define an outcome…”	_truth is absent	Call add_outcome() after the current add_data()
“Please define at least one metric…”	No metrics were selected	Call set_metrics()
Threshold error	Neither the score nor the evaluation call supplied a threshold	Store a threshold with add_score() or pass one to evaluate_metrics()
AUROC or AUPRC is NaN for a level	The level has one outcome class, null data, or another scikit-learn input problem	Inspect support and clean the level’s inputs
A declared category is a NaN row	The category has no observations	Keep it as an explicit placeholder or remove it from the pandas category list

See Exceptions and edge cases for exact behavior.