Schemas and result objects

Import configuration and result schemas from:

from model_auditor.schemas import (
    AuditorFeature,
    AuditorOutcome,
    AuditorScore,
    CalibrationEvaluation,
    ConditionalThreshold,
    FeatureEvaluation,
    LevelEvaluation,
    LevelMetric,
    InferenceConfig,
    ScoreEvaluation,
    ThresholdSpec,
)

The classes are mutable dataclasses. Snippets that use field(default_factory=...) assume field from dataclasses.

Configuration schemas

InferenceConfig

InferenceConfig controls confidence level, automatic versus bootstrap interval selection, Wilson versus exact binomial rates, IID/stratified/cluster resampling, cluster ID, local random seed, missing-feature policy, and minimum valid-resample thresholds. It is also exported from model_auditor.

AuditorFeature

@dataclass
class AuditorFeature:
    name: str
    label: str | None = None

• name: DataFrame column and internal dictionary key.
• label: human-readable feature text.

ConditionalThreshold

@dataclass
class ConditionalThreshold:
    feature: str
    levels: dict[Any, float]
    default: float | None = None

• feature: column used for row-level threshold selection.
• levels: native feature-value to threshold mapping.
• default: optional fallback for unmapped or null feature values.

Validation occurs during evaluation, not dataclass construction.

ThresholdSpec

ThresholdSpec = float | ConditionalThreshold

The runtime resolver also attempts float(value) for scalar candidates, then rejects non-finite results.

AuditorScore

@dataclass
class AuditorScore:
    name: str
    label: str | None = None
    threshold: ThresholdSpec | None = None

threshold=None is valid registration state, but threshold-dependent evaluation later requires a stored or call-time specification.

AuditorOutcome

@dataclass
class AuditorOutcome:
    name: str
    mapping: dict[Any, int] | None = None

Auditor uses the dataclass to call add_outcome() and materialize _truth.

LevelMetric

@dataclass
class LevelMetric:
    name: str
    label: str
    score: float | int
    interval: tuple[float, float] | None = None
    status: str = "ok"
    interval_status: str = "not_requested"
    interval_method: str | None = None
    denominator: int | None = None
    valid_resamples: int = 0
    requested_resamples: int = 0
    nonfinite_resamples: int = 0
    direction: str | None = None
    parameters: dict = field(default_factory=dict)

This is the atomic ordinary result. Diagnostic fields explain undefined estimates and withheld intervals without guessing from NaN alone.

• score is numeric.
• interval stores lower and upper bounds for eligible observed metrics.
• Count metrics and categorical placeholders generally have no interval.

LevelEvaluation

@dataclass
class LevelEvaluation:
    name: str
    metrics: dict[str, LevelMetric] = field(default_factory=dict)

LevelEvaluation.update

update(
    metric_name: str,
    metric_label: str,
    metric_score: float,
) -> None

Creates or replaces one metric under metrics[metric_name].

LevelEvaluation.update_intervals

update_intervals(
    metric_intervals: dict[str, tuple[float, float]],
) -> None

Sets intervals on existing metrics.

LevelEvaluation.to_dataframe

to_dataframe(
    n_decimals: int = 3,
    add_index: bool = False,
    metric_labels: bool = False,
) -> pd.DataFrame

Returns one display-formatted row.

add_index is unused at this level in v0.1.16.

Formatting:

• CI metric: score (lower, upper);
• float without CI: fixed decimals;
• integer: comma-delimited integer text.

LevelEvaluation.style_dataframe

style_dataframe(
    n_decimals: int = 3,
    metric_labels: bool = False,
    include_count_metrics: bool = False,
    low_color: str = "#f8d7da",
    medium_color: str = "#fff3cd",
    high_color: str = "#d4edda",
    rank: bool = False,
) -> pd.io.formats.style.Styler

Styling is neutral unless rank=True. A single-row table has no useful comparative tier context even though the method is available.

FeatureEvaluation

@dataclass
class FeatureEvaluation:
    name: str
    label: str
    levels: dict[str, LevelEvaluation] = field(default_factory=dict)
    excluded_n: int = 0
    total_n: int = 0

FeatureEvaluation.update

update(
    metric_name: str,
    metric_label: str,
    data: dict[str, float],
) -> None

Adds one metric across a level-to-score mapping.

FeatureEvaluation.update_intervals

update_intervals(
    level_name: str,
    metric_intervals: dict[str, tuple[float, float]],
) -> None

Updates one level’s intervals.

FeatureEvaluation.to_dataframe

to_dataframe(
    n_decimals: int = 3,
    add_index: bool = False,
    metric_labels: bool = False,
) -> pd.DataFrame

Returns one display-formatted row per level.

With add_index=True, the feature label becomes an outer index level.

FeatureEvaluation.style_dataframe

Has the same arguments as LevelEvaluation.style_dataframe(), applied across the feature’s levels.

ScoreEvaluation

@dataclass
class ScoreEvaluation:
    name: str
    label: str
    features: dict[str, FeatureEvaluation] = field(default_factory=dict)
    metadata: dict = field(default_factory=dict)

ScoreEvaluation.to_dataframe

to_numeric_dataframe() returns unrounded long-form estimates, support, diagnostics, stable identifiers, and a copy of provenance in frame.attrs["metadata"].

to_dataframe(
    n_decimals: int = 3,
    add_index: bool = False,
    metric_labels: bool = False,
) -> pd.DataFrame

Returns a display-formatted table with feature label and level name as the row MultiIndex.

With add_index=True, the score label becomes an additional outer index.

ScoreEvaluation.style_dataframe

style_dataframe(
    n_decimals: int = 3,
    metric_labels: bool = False,
    include_count_metrics: bool = False,
    low_color: str = "#f8d7da",
    medium_color: str = "#fff3cd",
    high_color: str = "#d4edda",
    rank: bool = False,
) -> pd.io.formats.style.Styler

Styling is neutral unless rank=True. Ranking is computed within each feature, not across unrelated feature dimensions.

ScoreEvaluation.plot_metric_intervals

plot_metric_intervals(
    metric: str,
    feature_names: list[str] | None = None,
    include_overall: bool = True,
    rotate_plots: bool = False,
    include_sample_size: bool = True,
    include_class_balance: bool = True,
) -> dict[str, tuple[Figure, Axes]]

See Plotting API for behavior and errors.

Ordering and placeholders

For pandas categorical features:

• levels dictionary order follows declared category order;
• all declared categories appear;
• an unobserved category receives a LevelEvaluation;
• every selected metric is present with score=NaN; and
• intervals remain None.

The result hierarchy preserves that order in tables and plots that consume the hierarchy.

For non-categorical features, ordinary metric and error evaluation convert level values to strings and use pandas’ default sorted group order. This is normally lexicographic order of the normalized string keys, not first-appearance order. Score-distribution plots are different: they preserve first appearance for non-categorical levels.

Styling helper behavior

The ordinary style methods classify known count names by exact, case-insensitive matching. TPR, TNR, FPR, and FNR are not misclassified as counts.

Each metric’s direction metadata controls ranking. A direction of "none" suppresses performance color.

Tiers are based on strict percentile rank:

lower third
middle third
upper third

NaN values receive no tier style.