Plot metric confidence intervals

ScoreEvaluation.plot_metric_intervals() turns stored analytic or resampled intervals into a separate Matplotlib figure for each selected feature.

Install Matplotlib

Matplotlib is optional and is not a core Model Auditor dependency.

python -m pip install matplotlib

For a non-interactive environment, set the backend before importing pyplot:

import matplotlib

matplotlib.use("Agg")

Evaluate with CI and support metrics

Interval plots require a CI-eligible metric evaluated with a positive bootstrap count.

Include count metrics when you want the default sample-size and class-balance annotations to contain values rather than NA.

from model_auditor.metrics import (
    Sensitivity,
    Specificity,
    nData,
    nNegative,
    nPositive,
)

auditor.set_metrics(
    [
        Sensitivity(),
        Specificity(),
        nData(),
        nPositive(),
        nNegative(),
    ]
)

results = auditor.evaluate_metrics(
    score_name="risk_score",
    n_bootstraps=1000,
)

Plot one metric

The selector can be the metric’s internal name or display label:

plots = results.plot_metric_intervals(
    metric="sensitivity",
)

Equivalent label form:

plots = results.plot_metric_intervals(
    metric="Sensitivity",
)

Each entry is a (Figure, Axes) tuple:

fig, ax = plots["region"]
fig.savefig(
    "region-sensitivity.png",
    dpi=180,
    bbox_inches="tight",
)

Select and order features

plots = results.plot_metric_intervals(
    metric="sensitivity",
    feature_names=["region", "age_group"],
)

The output dictionary follows the requested order.

The synthetic overall feature is never emitted as a standalone figure. By default, its Overall level is prepended inside every subgroup figure when it has a usable interval.

Disable the comparator:

plots = results.plot_metric_intervals(
    metric="sensitivity",
    include_overall=False,
)

Change orientation

Horizontal point-and-whisker plots are the default:

horizontal = results.plot_metric_intervals(
    metric="sensitivity",
    rotate_plots=False,
)

Rotate to put feature levels on the x-axis:

vertical = results.plot_metric_intervals(
    metric="sensitivity",
    rotate_plots=True,
)

Control annotations

Default annotations include:

N: subgroup count (% of overall)
N Pos: positive count (% positive within subgroup)

Turn either off:

plots = results.plot_metric_intervals(
    metric="sensitivity",
    include_sample_size=False,
    include_class_balance=False,
)

Counts are read from n, n_pos, and n_neg metrics when available, or derived from all four confusion-count metrics. Missing support produces NA text rather than an exception.

Customize the returned axes

fig, ax = plots["region"]

ax.set_xlim(0.0, 1.0)
ax.axvline(
    0.80,
    linestyle="--",
    label="Target",
)
ax.legend()

fig.tight_layout()
fig.savefig("region-sensitivity-with-target.png", dpi=180)

Understand exclusions and errors

A level is omitted when:

• its score is NaN;
• it has no interval; or
• either interval bound is NaN.

This automatically removes unobserved categorical placeholders and count metrics.

The method raises ValueError when a selected feature has no plottable level. If you evaluated with n_bootstraps=None, rerun with a positive count before plotting.

Infinite interval bounds are not rejected by the plottability check. Inspect sparse error-like distributions before using automatic axes limits.

See Plotting API for the exact signature.