ErrorEvaluation output

Import:

from model_auditor.schemas import ErrorEvaluation

Auditor.evaluate_errors() returns this dataclass. The signature below assumes field from dataclasses.

Error-metric protocol and built-in ratio

The error-metric module exposes:

from model_auditor.error_metrics import AuditorErrorMetric, OddsRatio

The structural protocol is:

AuditorErrorMetric
  name: str
  label: str
  ci_eligible: bool
  compute(
      group_count: int,
      group_total: int,
      full_count: int,
      full_total: int,
  ) -> float

OddsRatio has name="odds_ratio", label="Odds Ratio", and ci_eligible=True. Its compute() method implements the canonical level-versus-rest 2×2 ratio described in Error-group analysis.

evaluate_errors() uses OddsRatio by default. Pass error_metric= to supply a compatible count-based enrichment metric. The legacy wide export is specialized for OddsRatio; use to_numeric_dataframe() for generic metric exports.

Fields

@dataclass
class ErrorEvaluation:
    name: str
    label: str
    threshold: ThresholdSpec
    groups: dict[str, ScoreEvaluation] = field(default_factory=dict)
    global_total_n: int = 0
    support_data: dict = field(default_factory=dict)
    metadata: dict = field(default_factory=dict)

groups

The stable keys are inserted in this order:

tp
tn
fp
fn

Each value is a ScoreEvaluation whose:

• name is the group key;
• label is the uppercase group name; and
• features contain overall plus every registered feature.

Every level has one odds_ratio metric.

to_numeric_dataframe

Returns long-form enrichment estimates with support, status, interval method, resample diagnostics, and provenance in frame.attrs["metadata"].

threshold

Stores the resolved scalar or ConditionalThreshold used for the analysis.

global_total_n

Number of rows in the full internal error-analysis slice before dropping null values from individual feature columns.

support_data

Nested shape:

group
  → feature name
    → level name
      → n
      → pct_overall
      → pct_group

pct_overall uses global_total_n. pct_group uses the size of the current confusion group after dropping null values for the feature. When that group is empty, pct_group is reported as 0.0; its odds ratios are generally undefined and remain NaN.

to_dataframe

to_dataframe(
    n_decimals: int = 3,
    metric_labels: bool = False,
) -> pd.DataFrame

Returns a numeric DataFrame.

n_decimals is ignored in v0.1.16.

Row index

Two levels:

feature label
level name

The row order follows the first confusion group’s feature and level ordering.

Column index

Two levels:

section
metric

Overall section

Column	Meaning
("Overall", "N")	TP + TN + FP + FN count for the level
("Overall", "% overall")	level N divided by global_total_n
("Overall", "N_pos")	TP + FN
("Overall", "N_neg")	TN + FP
("Overall", "Pos %")	N_pos / (N_pos + N_neg)

TP, TN, FP, and FN sections

With metric_labels=False:

Subcolumn	Meaning
N	level rows in the group
% overall	group-level N divided by global total
% group	group-level N divided by group total
odds_ratio	point estimate
odds_ratio_ci_lower	lower percentile bound
odds_ratio_ci_upper	upper percentile bound

With metric_labels=True, the last three become:

Odds Ratio
OR 95% CI Lower
OR 95% CI Upper

The overall / Overall row has NaN odds-ratio and interval cells because every row belongs to that level and no comparator population exists.

style_dataframe

style_dataframe(
    n_decimals: int = 3,
    metric_labels: bool = False,
    include_count_metrics: bool = False,
    low_color: str = "#f8d7da",
    medium_color: str = "#fff3cd",
    high_color: str = "#d4edda",
) -> pd.io.formats.style.Styler

Behavior:

• builds the numeric DataFrame;
• hides separate CI-bound columns;
• formats an OR as point (lower, upper) when both bounds exist;
• displays an em dash for NaN;
• formats N columns as integers;
• keeps all enrichment odds-ratio cells neutral.

include_count_metrics is retained for API consistency but is unused by this method.

Interval effect

When n_bootstraps=None:

• OR is the exact observed contingency-table ratio;
• CI cells are NaN.

With default IID automatic inference, finite observed odds ratios receive conditional exact intervals. Other designs use diagnosed resampling. The stored point always remains the observed contingency-table ratio. An undefined observed ratio receives no interval.

Sparse outcomes

Possible point or bound values include:

0.0
finite positive ratio
+Infinity
NaN

The stable release applies no continuity correction.