Auditor API

Import:

from model_auditor import Auditor

Constructor

Auditor(
    data: pd.DataFrame | None = None,
    features: list[AuditorFeature] | None = None,
    scores: list[AuditorScore] | None = None,
    outcome: AuditorOutcome | None = None,
    metrics: list[AuditorMetric] | None = None,
) -> None

Parameters

Parameter	Meaning
data	Source DataFrame. Stored as a copy.
features	Optional configuration dataclasses, added in list order.
scores	Optional score dataclasses, added in list order.
outcome	Optional outcome dataclass. Requires data because it creates _truth.
metrics	Optional selected metric objects.

Example:

from model_auditor import Auditor
from model_auditor.metrics import AUROC, Sensitivity
from model_auditor.schemas import (
    AuditorFeature,
    AuditorOutcome,
    AuditorScore,
)

auditor = Auditor(
    data=data,
    features=[
        AuditorFeature(name="region", label="Region"),
    ],
    scores=[
        AuditorScore(
            name="risk_score",
            label="Risk score",
            threshold=0.5,
        ),
    ],
    outcome=AuditorOutcome(name="outcome"),
    metrics=[
        Sensitivity(),
        AUROC(),
    ],
)

Public state

Attribute	Type	Meaning
data	`pd.DataFrame	None`
features	dict[str, AuditorFeature]	Registered feature configurations
scores	dict[str, AuditorScore]	Registered score configurations
metrics	list[AuditorMetric]	Selected metric objects

The outcome is represented by _truth in data; v0.1.16 does not expose a separate public outcome attribute.

add_data

add_data(data: pd.DataFrame) -> None

Replaces internal data with a new copy.

Calling it after add_outcome() removes the previously derived _truth column unless that column already exists in the new source. Re-register the outcome after replacing data.

add_feature

add_feature(
    name: str,
    label: str | None = None,
) -> None

Registers a subgroup column under features[name].

• name is the DataFrame column.
• label is used in result indexes and plot titles; the name is used when no label exists.
• Reusing a name replaces the previous configuration.
• Column existence is validated later, when the feature is accessed.

Features are evaluated independently, not as automatic intersections.

add_score

add_score(
    name: str,
    label: str | None = None,
    threshold: ThresholdSpec | None = None,
) -> None

Registers a continuous score under scores[name].

ThresholdSpec is:

float | ConditionalThreshold

Reusing a name replaces the previous score configuration.

No score values are converted or validated at registration time.

add_outcome

add_outcome(
    name: str,
    mapping: dict[Any, int] | None = None,
) -> None

Creates the internal _truth column.

Without a mapping:

auditor.data["_truth"] = auditor.data[name]

With a mapping:

auditor.data["_truth"] = auditor.data[name].map(mapping)

Raises ValueError when no data has been added.

The method does not validate that mapped values are only 0 and 1 or that the mapping is complete.

set_metrics

set_metrics(
    metrics: list[AuditorMetric],
) -> None

Replaces the selected metric list. Metric names should be unique.

add_intersection

add_intersection(
    name: str,
    features: list[str],
    label: str | None = None,
) -> None

Creates and registers a joint subgroup from distinct existing feature columns. Components are JSON-encoded to avoid ambiguous delimiter collisions. Rows with any missing component remain missing in the intersection.

Comparisons, calibration, and decisions

compare_scores() calculates paired metric differences or ratios between two registered scores on the same rows. compare_groups() compares each observed level of a registered feature with a named reference level. Both recompute metrics inside shared resamples and return ScoreEvaluation.

evaluate_calibration() returns fixed equal-width reliability bins plus a ScoreEvaluation summary containing Brier score, log loss, calibration intercept, and calibration slope. decision_curve() returns descriptive net benefit, act-all, act-none, selection-rate, and sample-size columns across probability thresholds. These probability APIs require finite scores in [0, 1] and do not require a stored decision threshold.

See Compare models and groups and Evaluate calibration and decisions.

optimize_score_threshold

optimize_score_threshold(
    score_name: str,
) -> float

Builds an ROC curve and returns the threshold maximizing the Youden criterion:

TPR - FPR

Behavior:

• requires at least one registered score;
• requires data and _truth;
• validates finite score values and binary truth;
• uses scikit-learn roc_curve() with its default drop_intermediate=True;
• considers finite observed score thresholds;
• emits UserWarning with the selected threshold;
• does not mutate the score; and
• rejects an unknown score with ValueError and lists available scores.

optimize_score_threshold_for_target

optimize_score_threshold_for_target(
    score_name: str,
    target: float,
    metric: Literal["sensitivity", "specificity"] = "sensitivity",
) -> float

Returns a finite ROC threshold satisfying:

selected metric >= target

Validation:

• metric must be exactly "sensitivity" or "specificity";
• target must be in [0.0, 1.0];
• a score, data, and _truth must exist; and
• the score name must be registered.

Tie breaking:

• sensitivity: first feasible ROC index, corresponding to the highest feasible threshold;
• specificity: last feasible ROC index, corresponding to the lowest feasible threshold.

The ROC call uses drop_intermediate=False.

Raises ValueError with the achievable finite metric range when no finite threshold satisfies the target. Emits UserWarning on success and does not mutate the score.

evaluate_metrics

evaluate_metrics(
    score_name: str,
    threshold: ThresholdSpec | None = None,
    n_bootstraps: int | None = 1000,
    *,
    inference: InferenceConfig | None = None,
    cohort: str | None = None,
) -> ScoreEvaluation

Evaluates every selected metric for:

1. the automatic overall / Overall level; and
2. every level of every registered feature.

No registered feature is required. With an empty feature registry, the result contains only the automatic overall feature.

Requirements:

• data exists;
• _truth exists;
• at least one metric is selected;
• score_name is registered; and
• an effective stored or call-time threshold exists when a selected metric uses threshold-derived inputs.

Threshold handling:

• call-time threshold overrides the score’s stored threshold;
• scalar and conditional values must resolve to finite floats;
• binary prediction uses score >= threshold.

n_bootstraps=None disables intervals. A positive integer requests intervals under InferenceConfig; automatic IID inference uses analytic intervals where available and diagnosed percentile resampling otherwise.

Returns a new ScoreEvaluation. The internal source DataFrame is not replaced by the evaluation slice. The feature loop is displayed through tqdm; v0.1.16 does not expose a method argument to disable that progress output.

evaluate_errors

evaluate_errors(
    score_name: str,
    threshold: ThresholdSpec | None = None,
    n_bootstraps: int | None = 1000,
    *,
    inference: InferenceConfig | None = None,
    cohort: str | None = None,
    error_metric: AuditorErrorMetric | None = None,
) -> ErrorEvaluation

Calculates level-versus-rest odds ratios for TP, TN, FP, and FN groups.

No registered feature is required. With none configured, each confusion group contains only the synthetic overall feature; its odds ratio is undefined because no comparator population exists, while support counts remain available in the wide export.

Requirements are the same as ordinary evaluation except no metrics need to be selected.

The result records the resolved threshold.

Scores always remain observed-table odds ratios. Default IID automatic inference uses conditional exact intervals; other designs use diagnosed resampling.

plot_score_distributions

plot_score_distributions(
    score_name: str,
    feature_names: list[str] | None = None,
    bins: int | str = 30,
    density: bool = True,
    split_classes: bool = False,
) -> dict[str, tuple[Figure, np.ndarray]]

Creates one Matplotlib figure per selected feature and one stacked axis per observed level.

Requirements:

• Matplotlib is installed;
• data exists;
• at least one feature is registered;
• the score is registered; and
• every selected feature has at least one non-null feature/score row.

Behavior:

• feature_names=None uses all features in registration order;
• an explicit list controls selection and output order; an empty list returns {};
• shared bin edges are computed once per feature;
• categorical levels follow declared order among observed levels;
• plain levels follow first appearance order;
• feature or score null rows are excluded;
• density mode normalizes each level independently;
• class splitting overlays negative and positive distributions, normalizing each nonempty class separately; and
• the axes return is always a one-dimensional NumPy array.

The method does not require selected metrics. It requires a registered binary, complete outcome only when split_classes=True.

State-management notes

Internal evaluation columns use _truth, _pred, _binary_pred, overall, tp, tn, fp, and fn. Do not register ordinary features with these names; see Exceptions and edge cases.

• Public setup methods return None; do not chain them.
• Feature and score dictionaries preserve registration order.
• add_data() copies; HierarchyPlotter.set_data() is a separate API with different storage behavior.
• A call-time threshold affects only that call.
• Optimized thresholds must be assigned or passed explicitly.