Error-group analysis

Ordinary subgroup metrics ask how a model performs within a level. Error-group analysis asks how a level is represented within a confusion group.

These are related but different views.

Confusion groups

After thresholding, every valid binary row belongs to one group:

Group	Truth	Prediction
TP	1	1
TN	0	0
FP	0	1
FN	1	0

For every registered feature level, Model Auditor compares group membership between that level and all other observed levels combined.

Contingency table

For a chosen level and group:

	In group	Not in group
Level	a	b
Not level	c	d

The canonical odds ratio is:

OR = (a × d) / (b × c)

Equivalent interpretation:

odds of group membership among level rows
divided by
odds of group membership among non-level rows

Interpretation by group

FP and FN

An OR above one means the level has higher odds of appearing in that error group than all other levels combined. These are usually the most direct error-concentration signals.

TP and TN

An OR above one means the level is over-represented in a correct-prediction group. This can reflect prevalence, subgroup size, score separation, threshold policy, or other factors. It is not a standalone performance ranking.

Support is essential

The same ratio can arise from very different counts. Read:

• level N;
• group N;
• percentage of the full dataset;
• percentage of the confusion group;
• positive-class fraction; and
• interval width.

A large finite or infinite ratio from one or two observations should not be treated like a stable large-cohort estimate.

Boundary cases

Unobserved level

If full_count == 0, there is no level population and the ratio is NaN.

No comparator population

The automatic overall / Overall level contains every row, so there is no non-level population. Its odds ratio is NaN.

Level absent from the group

When a == 0 and the denominator is positive, the ratio is 0.0.

Sparse zero denominator

When b × c == 0:

• 0 / 0 is indeterminate and becomes NaN;
• a nonzero numerator divided by zero becomes positive infinity.

No continuity correction is applied.

Interval meaning

Without bootstrap, the point is the direct ratio from the observed contingency table.

With default IID automatic inference, odds ratios receive conditional exact intervals. Other inference designs use diagnosed resampling from the full evaluation slice. In every case the point remains the direct ratio from the observed contingency table. No continuity correction is applied.

Causal and fairness limits

The ratio is descriptive. It does not establish:

• why a level is over-represented;
• whether the feature causes the error;
• whether group definitions are exchangeable;
• whether a disparity is statistically significant;
• whether a threshold policy is lawful or fair; or
• whether intervention on the feature would change performance.

Use it to locate patterns that deserve validation, not as an automatic verdict.