Getting started

This walkthrough installs the documented stable release, evaluates one score overall and by region, and verifies exact known results.

Before you begin

You need:

• Python 3.10 or later;
• a virtual environment or another isolated Python environment; and
• a pandas DataFrame with a continuous score, a binary outcome, and at least one subgroup feature.

The package metadata lists Python 3.10, 3.11, and 3.12 classifiers. The core package does not require Matplotlib or Plotly.

Install the stable release

python -m pip install model-auditor==0.1.16

Verify the import:

python -c "from model_auditor import Auditor; print(Auditor.__name__)"

Expected output:

Auditor

Create a small evaluation dataset

first_audit.py

import pandas as pd

from model_auditor import Auditor
from model_auditor.metrics import AUROC, Sensitivity, Specificity, nData

data = pd.DataFrame(
    {
        "region": [
            "North",
            "North",
            "North",
            "North",
            "South",
            "South",
            "South",
            "South",
        ],
        "risk_score": [0.90, 0.75, 0.65, 0.20, 0.85, 0.55, 0.40, 0.10],
        "outcome": [1, 1, 0, 0, 1, 0, 1, 0],
    }
)

auditor = Auditor()
auditor.add_data(data)
auditor.add_feature(name="region", label="Region")
auditor.add_score(name="risk_score", label="Risk score", threshold=0.5)
auditor.add_outcome(name="outcome")
auditor.set_metrics(
    [
        Sensitivity(),
        Specificity(),
        AUROC(),
        nData(),
    ]
)

results = auditor.evaluate_metrics(
    score_name="risk_score",
    n_bootstraps=None,
)

print(results.to_dataframe(metric_labels=True))

Run it:

python first_audit.py

The values should correspond to this table:

                 Sensitivity Specificity  AUROC  N
Overall Overall        0.750       0.500  0.875  8
Region  North          1.000       0.500  1.000  4
        South          0.500       0.500  0.750  4

The exact whitespace is pandas-dependent, but the row labels and values should match.

What happened

The configuration established five pieces of state:

1. add_data() copied the source DataFrame into the auditor.
2. add_feature() registered region as a subgroup dimension.
3. add_score() registered a continuous model score and a scalar decision threshold.
4. add_outcome() created the internal binary truth column.
5. set_metrics() selected the metrics computed for every level.

evaluate_metrics() then added an automatic Overall row and evaluated each observed region. Passing n_bootstraps=None made this first run fast and deterministic.

Threshold-independent and threshold-dependent metrics

Sensitivity and specificity use binary predictions produced by risk_score >= 0.5. AUROC uses the original continuous scores and does not change when only the decision threshold changes.

Inspect a raw value

The DataFrame is intended for display. Use the nested result objects when you need a numeric value:

north_sensitivity = (
    results.features["region"]
    .levels["North"]
    .metrics["sensitivity"]
    .score
)

assert north_sensitivity == 1.0

Next steps

• Add confidence intervals and more metrics in Evaluate subgroup performance.
• Learn how thresholds are resolved in Thresholds and predictions.
• See every built-in metric in Built-in metrics.
• Review the exact result hierarchy in Schemas and result objects.