Skip to content
Model Auditor logoModel Auditor

Analyze confusion-group representation

evaluate_errors() answers a different question from ordinary subgroup metrics:

How strongly is each feature level represented in a particular confusion-matrix group, relative to all other levels combined?

The stable release uses a canonical 2×2 odds ratio for every level in the true-positive, true-negative, false-positive, and false-negative groups.

Configure data, features, score, and outcome

Section titled “Configure data, features, score, and outcome”

Error analysis does not require set_metrics().

from model_auditor import Auditor


auditor = Auditor()
auditor.add_data(data)
auditor.add_feature(name="region", label="Region")
auditor.add_feature(name="age_group", label="Age group")
auditor.add_score(name="risk_score", threshold=0.5)
auditor.add_outcome(name="outcome")

Start with exact observed odds ratios

Section titled “Start with exact observed odds ratios”

Disable bootstrapping while checking counts and contingency-table logic:

errors = auditor.evaluate_errors(
    score_name="risk_score",
    n_bootstraps=None,
)

For a feature level and one confusion group, the ratio is:

OR = (a × d) / (b × c)

where:

  • a is the number of level rows in the group;
  • b is the number of level rows outside the group;
  • c is the number of non-level rows in the group; and
  • d is the number of non-level rows outside the group.

Read one value directly:

north_fn_or = (
    errors.groups["fn"]
    .features["region"]
    .levels["North"]
    .metrics["odds_ratio"]
    .score
)

Interpret the direction

Section titled “Interpret the direction”
  • OR = 1 means equal odds of group membership for the level and all other levels combined.
  • OR > 1 means the level is over-represented in that group.
  • OR < 1 means the level is under-represented.
  • OR = 0 can occur when the level exists in the dataset but has no rows in the group.
  • OR = +∞ can occur in sparse tables with a zero denominator and nonzero numerator.
  • NaN represents an undefined comparison, including an unobserved level or a level covering the full comparator population.

Direction is contextual. A high ratio in FP or FN identifies concentration in an error group. A high ratio in TP or TN identifies concentration in a correct group; it is not automatically “better” without considering support, prevalence, and the operating policy.

Export the wide numeric table

Section titled “Export the wide numeric table”
error_table = errors.to_dataframe(metric_labels=True)
print(error_table)

Rows use a (feature, level) MultiIndex. Columns use a (section, metric) MultiIndex.

The Overall section contains:

  • N;
  • % overall;
  • N_pos;
  • N_neg; and
  • Pos %.

Each TP, TN, FP, and FN section contains:

  • group N;
  • % overall;
  • % group;
  • odds ratio; and
  • lower and upper CI columns.

Unlike performance-result exports, this DataFrame is numeric and ready for further pandas calculations.

Add interval uncertainty

Section titled “Add interval uncertainty”
from model_auditor import InferenceConfig


bootstrapped_errors = auditor.evaluate_errors(
    score_name="risk_score",
    n_bootstraps=1000,
    inference=InferenceConfig(random_state=20260831),
)

Sparse samples can produce zero, infinity, NaN, or very wide intervals. Always inspect the support columns before interpreting a ratio.

Display a compact notebook table

Section titled “Display a compact notebook table”
display(
    bootstrapped_errors.style_dataframe(
        n_decimals=3,
        metric_labels=True,
    )
)

The styled view folds CI bounds into each odds-ratio cell and removes the separate bound columns. Enrichment odds ratios are neutral because they have no universal better-or-worse direction.

Validate one ratio manually

Section titled “Validate one ratio manually”
feature = "region"
level = "North"
group = "fn"


support = errors.support_data
a = int(support[group][feature][level]["n"])


group_total = sum(
    int(level_data["n"])
    for level_data in support[group][feature].values()
)


# The OR calculation drops only rows whose value is null for this feature.
# It does not derive the full feature population by summing confusion groups.
feature_rows = data.loc[data[feature].notna()]
full_total = len(feature_rows)
full_count = int(
    (feature_rows[feature].astype(str) == level).sum()
)


b = full_count - a
c = group_total - a
d = (full_total - full_count) - c


numerator = a * d
denominator = b * c


if full_count == 0 or full_total - full_count == 0:
    manual_or = float("nan")
elif denominator == 0 and numerator == 0:
    manual_or = float("nan")
elif denominator == 0:
    manual_or = float("inf")
else:
    manual_or = numerator / denominator


print(manual_or)

The verifier assumes data is the same source used by the auditor. For each feature, the OR contingency table drops only rows whose value is null for that feature; % overall instead uses global_total_n. This distinction also matters when rows—for example, rows with null truth—do not enter any confusion group.

Next steps

Section titled “Next steps”