Exceptions and edge cases

This page centralizes stable-release behavior that can otherwise look like a metric discrepancy.

Setup and evaluation errors

Condition	Public surface	Behavior
Outcome added before data	add_outcome()	ValueError
No data	evaluation or plotting	ValueError requesting add_data()
No _truth	evaluation or optimization	ValueError requesting add_outcome()
No selected metrics	evaluate_metrics()	ValueError
Unknown score	evaluate_metrics(), evaluate_errors(), target optimizer, distributions	ValueError with available scores
No effective threshold	error evaluation or threshold-dependent metrics	ValueError
Unknown selected feature	interval or distribution plot	ValueError with available features
Missing Matplotlib	Matplotlib methods	ImportError with installation command
No hierarchy features	HierarchyPlotter.compile()	ValueError
No hierarchy data	hierarchy preparation	ValueError

Feature and score column existence is often deferred to pandas indexing, so missing columns can raise KeyError.

evaluate_metrics() and evaluate_errors() can run with no registered audit features because both create an automatic overall feature. An overall-only ScoreEvaluation cannot produce an interval plot: plot_metric_intervals() excludes the synthetic overall feature and then raises because no plottable feature remains. plot_score_distributions() instead requires at least one registered feature before plotting.

Threshold validation

Scalar and conditional threshold values are passed through float() and then checked with np.isfinite().

Rejected examples:

float("nan")
float("inf")
float("-inf")
object()

A conditional threshold also fails when:

• feature is an empty string;
• the feature column is missing from the evaluation data;
• an observed level has no mapping and no default;
• a condition value is null and no default exists; or
• any level/default threshold is non-finite or nonnumeric.

A condition feature is included in the evaluation slice automatically even when it is not registered as an audit feature.

Both optimizers require both truth classes and return finite thresholds. Target optimization also considers a finite all-negative endpoint when representable.

Outcome mapping

auditor.add_outcome(
    name="status",
    mapping={
        "case": 1,
        "control": 0,
    },
)

Series.map() converts unmapped values to null.

Validate before evaluation:

known = set(data["status"].dropna().unique())
mapped = {"case", "control"}

unknown = known.difference(mapped)
if unknown:
    raise ValueError(f"Unmapped outcomes: {sorted(unknown)}")

Evaluation enforces nonmissing binary _truth values after mapping. Unmapped or nonbinary outcomes are rejected before metrics are calculated.

Missing scores and truth values

Evaluation rejects missing or nonfinite selected scores and missing or nonbinary truth before metrics are calculated.

Missing feature values

For each registered feature, rows with a null value in that feature are removed from that feature’s calculation.

They still contribute to:

• the automatic overall row;
• other features whose values are not null; and
• global_total_n in error analysis.

Therefore, feature-level % overall values can sum below one.

Score-distribution plotting removes rows where either the current feature or score is null. Passing feature_names=[] to plot_score_distributions() returns an empty dictionary rather than raising.

Categorical placeholders

A pandas categorical feature retains all declared categories in metric and error tables.

An unobserved category:

• has zero support;
• receives NaN metric or odds-ratio scores;
• has no interval; and
• is omitted from interval and score-distribution plots.

Remove unused categories if placeholders are not desired:

data["group"] = data["group"].cat.remove_unused_categories()

Single-class levels

AUROC requires both truth classes. A level containing only positives or only negatives yields NaN when scikit-learn raises ValueError.

AUPRC uses average_precision_score; its behavior follows the installed scikit-learn version and can include warnings or boundary values for degenerate truth arrays.

Sensitivity, specificity, precision, F-scores, FPR, FNR, NPV, and MCC return NaN when their denominator is zero.

In error analysis, a completely empty confusion group reports 0.0 for % group support values, while its level-versus-rest odds ratios are generally NaN because both ratio terms collapse.

These conventions prevent division errors but do not make an undefined scientific quantity informative. Always report support.

Bootstrap edge cases

Use:

None

to disable bootstrap, or a positive integer to enable it.

A zero or boolean count is rejected; use None to disable intervals.

Bootstrap samples can:

• contain one outcome class;
• produce NaN ranking metrics;
• produce NaN or infinite odds ratios; and
• have interval bounds based only on non-NaN replicates.

Valid, requested, and nonfinite resample counts are recorded with interval status.

Progress output

evaluate_metrics() uses tqdm around its feature loop and can write a progress display in terminals, logs, or notebooks. There is no public per-call switch to disable it in v0.1.16. evaluate_errors() does not use that wrapper.

Randomness

Set a local seed with InferenceConfig:

from model_auditor import InferenceConfig

inference = InferenceConfig(random_state=42)

The global NumPy random state is not consumed.

Result formatting

Ordinary to_dataframe() methods format numeric values into strings. A failed numeric operation on the exported table is therefore usually a representation issue, not a source-metric issue.

ErrorEvaluation.to_dataframe() is numeric and ignores n_decimals.

Styling edge cases

• All equal values receive one consistent tier.
• NaN receives no style.
• Count recognition is exact and case-insensitive.
• FPR/FNR direction is inverted.
• Other custom lower-is-better metrics are not automatically inverted.
• Tier boundaries are relative thirds, not fixed numeric cutoffs.

Plotting limitations

• Interval plots require non-NaN stored intervals.
• An all-placeholder feature cannot produce an interval plot.
• Score distribution class splitting requires a registered complete binary outcome.
• Hierarchy score thresholds are not applied.
• Hierarchy callable aggregators receive DataFrames.
• Empty hierarchy definitions are not usable.
• Invalid pandas aggregator names or callables fail at compile time.

Metric and level identifier collisions

Metric name values must be unique within the selected list. Duplicate names are rejected before evaluation.

When metric_labels=True, display tables are built with labels as dictionary keys. Two distinct metrics with the same label can collapse into one output column even when their internal names differ. Keep both names and labels unique when all metrics must remain visible.

Feature labels become outer row-index values. Reusing one display label for multiple registered features can create duplicate (feature label, level) index keys when level names overlap, making selection, export, and styling ambiguous. Keep feature labels unique in report-oriented workflows.

Ordinary and error evaluation normalize feature-level keys to strings. Non-categorical columns are converted to strings before grouping and normally appear in pandas’ default lexicographically sorted group order, while categorical group keys are stringified during result assembly. Distinct native values with the same string representation—for example integer 1 and string "1"—can merge or overwrite one another. Normalize level values to unambiguous labels before evaluation.

Reserved internal column names

Avoid using these names for ordinary audit-feature columns:

_truth
_pred
_binary_pred
overall
tp
tn
fp
fn

add_outcome() writes _truth into the auditor’s internal DataFrame copy. Evaluation then writes _pred, _binary_pred, the synthetic overall column, and requested confusion indicators into a temporary slice. A registered feature with one of those names can therefore be overwritten or grouped by internal values rather than the source column. A pre-existing _truth column can also satisfy the outcome-presence check even when add_outcome() was never called.

The caller’s original DataFrame is still protected by add_data()’s copy, but the evaluation semantics can be wrong. Rename conflicting source columns and create truth through add_outcome() deliberately.

HierarchyPlotter also uses _temp_feature inside its copied compile-time DataFrame when multiple hierarchy items are valid at one level. Treat that name as reserved within hierarchy definitions.

Data replacement

add_data() replaces the stored DataFrame copy without replaying earlier outcome setup. Configure the outcome after the final data replacement.

Features, scores, and selected metrics remain registered unless replaced explicitly, so confirm that their column names still match the new DataFrame.