Model Auditor
Model Auditor is a Python library for evaluating fixed binary-classification predictions overall and across subgroups. It combines classification, ranking, probability, and decision metrics with explicit inference, confusion-group error analysis, paired comparisons, and visualization-ready outputs.
Use it when you already have one row per evaluated observation in a pandas DataFrame, including:
- a binary ground-truth outcome;
- one or more continuous score columns; and
- categorical or discrete columns that define subgroups.
What you can do
Section titled “What you can do”- Compare sensitivity, specificity, AUROC, AUPRC, precision, F-scores, MCC, error rates, and support counts across feature levels.
- Attach one scalar threshold to a score, override it for a single run, or apply different thresholds by feature level.
- Select a scalar operating point with the Youden index or a minimum sensitivity or specificity target.
- Configure Wilson, exact, IID, stratified, or cluster-resampled pointwise intervals with a local seed and diagnostics.
- Compare paired models or feature levels with metric differences or ratios.
- Evaluate probability calibration and descriptive decision net benefit.
- Measure how strongly each subgroup is represented in true-positive, true-negative, false-positive, and false-negative sets.
- Export unrounded long-form results with support and provenance, display tables, or neutral-by-default pandas
Stylerobjects. - Plot metric intervals and subgroup score distributions with Matplotlib.
- Generate node arrays for sunburst or treemap charts rendered by Plotly or another library.
- Define custom metrics over the package’s recognized truth, score, and confusion-input columns.
Choose a path
Section titled “Choose a path”| Your goal | Start here |
|---|---|
| Install the package and run one verified evaluation | Getting started |
| Compare model performance across subgroups | Evaluate subgroup performance |
| Compare two scores or feature levels | Compare models and groups |
| Check probability calibration or net benefit | Evaluate calibration and decisions |
| Choose an operating threshold | Optimize a scalar threshold |
| Apply policy-specific thresholds by region, cohort, or channel | Apply conditional thresholds |
| Find subgroups concentrated in false positives or false negatives | Analyze confusion-group representation |
| Prepare notebook tables or publication inputs | Export and style results |
| Understand exact method signatures and defaults | Reference |
| Work on the package or synchronize this site | Contributing |
Model Auditor v0.1.16 evaluates binary outcomes. It does not train models, choose protected attributes for you, determine whether a disparity is acceptable, or convert relative table colors into a fairness conclusion. Threshold optimization uses the data you provide, so operating-point selection and final evaluation should be separated when unbiased performance estimates matter.
See package status and compatibility and known edge cases before adopting the package in a production workflow.