ErrorEvaluation output

Import:

from model_auditor.schemas import ErrorEvaluation

Auditor.evaluate_errors() returns this dataclass. The signature below assumes field from dataclasses.

Error-metric protocol and built-in ratio

The error-metric module exposes:

from model_auditor.error_metrics import AuditorErrorMetric, OddsRatio

The structural protocol is:

AuditorErrorMetric
  name: str
  label: str
  ci_eligible: bool
  compute(
      group_count: int,
      group_total: int,
      full_count: int,
      full_total: int,
  ) -> float

OddsRatio has name="odds_ratio", label="Odds Ratio", and ci_eligible=True. Its compute() method implements the canonical level-versus-rest 2×2 ratio described in Error-group analysis.

evaluate_errors() uses OddsRatio by default. Pass error_metric= to supply a compatible count-based enrichment metric. The legacy wide export is specialized for OddsRatio; use to_numeric_dataframe() for generic metric exports.

Fields

@dataclass
class ErrorEvaluation:
    name: str
    label: str
    threshold: ThresholdSpec
    groups: dict[str, ScoreEvaluation] = field(default_factory=dict)
    global_total_n: int = 0
    metadata: dict = field(default_factory=dict)
    support_data: dict = field(default_factory=dict)

groups

The stable keys are inserted in this order:

tp
tn
fp
fn

Each value is a ScoreEvaluation whose:

• name is the group key;
• label is the uppercase group name; and
• features contain overall plus every registered feature.

With the default error metric, every level has one odds_ratio metric. A custom metric uses its own name; use the generic numeric export for it.

to_numeric_dataframe

Returns long-form enrichment estimates with support, status, interval method, resample diagnostics, and provenance in frame.attrs["metadata"].

threshold

Stores the resolved scalar or ConditionalThreshold used for the analysis.

global_total_n

Number of rows in the full internal error-analysis slice before dropping null values from individual feature columns.

support_data

Nested shape:

group
  → feature name
    → level name
      → n
      → pct_overall
      → pct_group

pct_overall uses global_total_n. pct_group uses the size of the current confusion group after applying the configured feature-missingness policy (dropping nulls by default). When that group is empty, pct_group is NaN; its odds ratios are generally undefined and remain NaN. All pct_ values are fractions, not 0–100 percentages.

to_dataframe

to_dataframe(
    n_decimals: int = 3,
    metric_labels: bool = False,
) -> pd.DataFrame

Returns a numeric DataFrame.

n_decimals is ignored in v0.1.16.

Row index

Two levels:

feature label
level name

The row order follows the first confusion group’s feature and level ordering.

Column index

Two levels:

section
metric

Overall section

Column	Meaning
("Overall", "N")	TP + TN + FP + FN count for the level
("Overall", "% overall")	level N divided by global_total_n
("Overall", "N_pos")	TP + FN
("Overall", "N_neg")	TN + FP
("Overall", "Pos %")	N_pos / (N_pos + N_neg)

TP, TN, FP, and FN sections

With metric_labels=False:

Subcolumn	Meaning
N	level rows in the group
% overall	group-level N divided by global total
% group	group-level N divided by group total
odds_ratio	point estimate
odds_ratio_ci_lower	lower interval bound
odds_ratio_ci_upper	upper interval bound

With metric_labels=True and the default 0.95 confidence level, the last three become:

Odds Ratio
OR 95% CI Lower
OR 95% CI Upper

The percentage in these labels follows InferenceConfig.confidence_level.

The overall / Overall row has NaN odds-ratio and interval cells because every row belongs to that level and no comparator population exists.

style_dataframe

style_dataframe(
    n_decimals: int = 3,
    metric_labels: bool = False,
    include_count_metrics: bool = False,
    low_color: str = "#f8d7da",
    medium_color: str = "#fff3cd",
    high_color: str = "#d4edda",
) -> pd.io.formats.style.Styler

Behavior:

• builds the numeric DataFrame;
• hides separate CI-bound columns;
• formats an OR as point (lower, upper) when both bounds exist;
• displays an em dash for NaN;
• formats N columns as integers;
• keeps all enrichment odds-ratio cells neutral.

include_count_metrics is retained for API consistency but is unused by this method.

Interval effect

When n_bootstraps=None:

• OR is the exact observed contingency-table ratio;
• CI cells are NaN.

With default IID automatic inference, defined observed odds ratios (including zero and positive infinity) receive conditional exact intervals. Other designs use diagnosed resampling. The stored point always remains the observed contingency-table ratio. An undefined observed ratio receives no interval.

Sparse outcomes

Possible point or bound values include:

0.0
finite positive ratio
+Infinity
NaN

The stable release applies no continuity correction.