Evaluation model

Model Auditor evaluates an already-generated continuous score against a binary outcome. It organizes the same calculation across a synthetic overall group and every registered feature level.

Configuration objects

An audit has four logical inputs:

Input	Role
Data	One pandas DataFrame containing rows to evaluate
Features	Columns that define independent subgroup dimensions
Scores	Continuous prediction columns, labels, and optional thresholds
Outcome	One binary truth column, optionally mapped from source values

Metrics are a fifth configuration layer: they declare which derived inputs they require and how to aggregate a feature-level slice.

Each call evaluates one score name. Registering multiple scores lets the same auditor evaluate them in separate calls, but the result object represents one score at a time.

State and order

Auditor stores a copy of the DataFrame. Public mutator methods update internal state and return None.

A safe setup order is:

add_data
  → add_feature, add_score
  → add_outcome
  → set_metrics
  → evaluate

add_outcome() requires current data because it creates an internal _truth column immediately. Replacing the data later removes that derived column, so call add_outcome() again after add_data().

Adding a feature or score under an existing name replaces that entry in its internal dictionary.

Threshold resolution

Before metric calculation, an evaluation:

1. selects the registered score;
2. chooses a call-time threshold when present, otherwise the score’s stored threshold;
3. validates finite scalar or conditional threshold values;
4. constructs one threshold per row; and
5. creates _binary_pred from score >= threshold.

Threshold-independent metric selections do not require an effective threshold. The auditor creates _binary_pred only when a selected input needs it.

Metric-input discovery

Each metric declares an inputs list. The auditor gathers the union of those names.

Two inputs are already present:

• _truth;
• _pred.

The stable metric-input layer can derive:

• tp;
• tn;
• fp; and
• fn.

Only required confusion columns are added for ordinary metric evaluation.

For example:

Sensitivity.inputs = ["tp", "fn"]
AUROC.inputs = ["_truth", "_pred"]
nData.inputs = []

Feature evaluation

The auditor creates a temporary overall column whose value is Overall for every row, then appends each registered feature.

Each feature is evaluated independently:

• missing feature values follow InferenceConfig.missing (exclude, include, or error), and exclusions are recorded;
• observed levels become LevelEvaluation objects;
• every selected metric receives the level’s DataFrame slice; and
• eligible metrics can receive bootstrap intervals.

The same row can contribute to several feature tables because features are not nested or intersected by default. Registering region and age_group produces separate marginal analyses, not every region-by-age combination.

Use add_intersection() when that is the intended grouping:

auditor.add_intersection(
    features=["region", "age_group"],
    name="region_age",
    label="Region and age group",
)

Categorical levels

A pandas categorical feature is special:

• declared category order becomes result order;
• observed categories receive calculated metrics;
• declared but unobserved categories receive NaN metric placeholders; and
• placeholders do not receive confidence intervals.

This makes report schemas stable across cohorts, but it also means a row can exist without supporting observations.

Non-categorical feature values are converted to strings and grouped with pandas’ default sorted-group behavior, so their ordinary metric and error-result order is normally lexicographic rather than first appearance. Categorical result keys are also stringified. Distinct native values that stringify identically can therefore collide; normalize labels deliberately. Use a categorical dtype when exact order is a contract.

Result assembly

The final hierarchy is:

ScoreEvaluation
  ├─ overall
  │    └─ Overall
  │         └─ metrics
  ├─ first feature
  │    ├─ first level
  │    │    └─ metrics
  │    └─ next level
  │         └─ metrics
  └─ next feature
       └─ ...

Metrics are keyed by their machine-readable name, not their label.

Performance characteristics

evaluate_metrics() wraps the feature loop in tqdm, so it emits a progress display even when called as a library. v0.1.16 has no public disable argument for that progress bar. evaluate_errors() does not use the same progress wrapper.

The main evaluation loop is synchronous. Bootstrap work is repeated for every observed feature level and eligible metric. Runtime therefore grows with:

• number of feature levels;
• number of eligible metrics;
• rows per level; and
• n_bootstraps.

Disable intervals for exploratory configuration checks, then enable them for the final intended metric set.

What the model does not provide

The stable evaluation pipeline does not:

• infer subgroup features;
• validate that a feature is legally or ethically appropriate;
• estimate intersections unless you supply one;
• train or calibrate a model;
• clean missing scores or outcomes globally;
• correct for multiple comparisons;
• compare subgroup estimates with a formal hypothesis test; or
• determine an acceptable disparity.

Those decisions remain part of the surrounding evaluation design.