Exceptions and edge cases

This page centralizes stable-release behavior that can otherwise look like a metric discrepancy.

Setup and evaluation errors

Condition	Public surface	Behavior
Outcome added before data	add_outcome()	ValueError
No data	evaluation or plotting	ValueError requesting add_data()
No _truth	evaluation or optimization	ValueError requesting add_outcome()
No selected metrics	evaluate_metrics()	ValueError
Unknown score	evaluate_metrics(), evaluate_errors(), target optimizer, distributions	ValueError with available scores
No effective threshold	error evaluation or threshold-dependent metrics	ValueError
Unknown selected feature	interval or distribution plot	ValueError with available features
Missing Matplotlib	Matplotlib methods	ImportError with installation command
No hierarchy features	HierarchyPlotter.compile()	ValueError
No hierarchy data	hierarchy preparation	ValueError

Feature and score column existence is often deferred to pandas indexing, so missing columns can raise KeyError.

evaluate_metrics() and evaluate_errors() can run with no registered audit features because both create an automatic overall feature. An overall-only ScoreEvaluation cannot produce an interval plot: plot_metric_intervals() excludes the synthetic overall feature and then raises because no plottable feature remains. plot_score_distributions() instead requires at least one registered feature before plotting.

Threshold validation

Scalar and conditional threshold values are passed through float() and then checked with np.isfinite().

Rejected examples:

float("nan")
float("inf")
float("-inf")
object()

A conditional threshold also fails when:

• feature is an empty string;
• the feature column is missing from the evaluation data;
• an observed level has no mapping and no default;
• a condition value is null and no default exists; or
• any level/default threshold is non-finite or nonnumeric.

A condition feature is included in the evaluation slice automatically even when it is not registered as an audit feature.

Both optimizers require both truth classes and return finite thresholds. Target optimization also considers a finite all-negative endpoint when representable.

Outcome mapping

auditor.add_outcome(
    name="status",
    mapping={
        "case": 1,
        "control": 0,
    },
)

Series.map() converts unmapped values to null.

Validate before evaluation:

known = set(data["status"].dropna().unique())
mapped = {"case", "control"}

unknown = known.difference(mapped)
if unknown:
    raise ValueError(f"Unmapped outcomes: {sorted(unknown)}")

Evaluation enforces nonmissing binary _truth values after mapping. Unmapped or nonbinary outcomes are rejected before metrics are calculated.

Missing scores and truth values

Evaluation rejects missing or nonfinite selected scores and missing or nonbinary truth before metrics are calculated.

Missing feature values

By default (InferenceConfig(missing="exclude")), rows with a null value in a feature are removed from that feature’s calculation. missing="include" uses the label (Missing); missing="error" rejects missing levels.

They still contribute to:

• the automatic overall row;
• other features whose values are not null; and
• global_total_n in error analysis.

Therefore, feature-level % overall values can sum below one.

Score-distribution plotting removes rows where either the current feature or score is null. Passing feature_names=[] to plot_score_distributions() returns an empty dictionary rather than raising.

Categorical placeholders

A pandas categorical feature retains all declared categories in metric and error tables.

An unobserved category:

• has zero support;
• receives NaN metric or odds-ratio scores;
• has no interval;
• has no interval marker (but is listed in interval diagnostic text); and
• is omitted from score-distribution histograms.

Remove unused categories if placeholders are not desired:

data["group"] = data["group"].cat.remove_unused_categories()

Single-class levels

AUROC explicitly returns NaN without both truth classes. AUPRC and AveragePrecision explicitly return NaN when there are no positive truth labels; they use non-interpolated average precision otherwise.

Sensitivity, specificity, precision, F-scores, FPR, FNR, NPV, and MCC return NaN when their denominator is zero.

In error analysis, a completely empty confusion group reports NaN for % group support values, while its level-versus-rest odds ratios are generally NaN because both ratio terms collapse.

These conventions prevent division errors but do not make an undefined scientific quantity informative. Always report support.

Bootstrap edge cases

Use:

None

to disable all intervals (including analytic ones), or a positive integer to request intervals. Auto analytic methods do not consume bootstrap resamples.

A zero or boolean count is rejected; use None to disable intervals.

Bootstrap samples can:

• contain one outcome class;
• produce NaN ranking metrics;
• produce NaN or infinite odds ratios; and
• have interval bounds based only on finite replicates, subject to the configured minimum count/fraction and degeneracy checks.

Valid, requested, and nonfinite resample counts are recorded with interval status.

Progress output

evaluate_metrics() uses tqdm around its feature loop and can write a progress display in terminals, logs, or notebooks. There is no public per-call switch to disable it in v0.1.16. evaluate_errors() does not use that wrapper.

Randomness

Set a local seed with InferenceConfig:

from model_auditor import InferenceConfig

inference = InferenceConfig(random_state=42)

The global NumPy random state is not consumed.

Result formatting

Ordinary to_dataframe() methods format numeric values into strings. A failed numeric operation on the exported table is therefore usually a representation issue, not a source-metric issue.

ErrorEvaluation.to_dataframe() is numeric and ignores n_decimals.

Styling edge cases

• With rank=True, all-equal eligible values receive the middle-tier color.
• NaN receives no style.
• Count recognition is exact and case-insensitive.
• FPR/FNR direction is inverted.
• Custom direction="lower" metrics are also inverted when rank=True; missing/"none" directions stay neutral.
• Tier boundaries are relative thirds, not fixed numeric cutoffs.

Plotting limitations

• Interval markers require finite estimates and finite, ordered bounds.
• A feature with no usable intervals returns a diagnostic figure.
• Score distribution class splitting requires a registered complete binary outcome.
• Hierarchy score thresholds are not applied.
• Hierarchy callable aggregators receive DataFrames.
• Empty hierarchy definitions compile to a root-only node.
• Invalid pandas aggregator names or callables fail at compile time.

Metric and level identifier collisions

Metric name values must be unique within the selected list. Duplicate names are rejected before evaluation.

When metric_labels=True, display tables are built with labels as dictionary keys. Two distinct metrics with the same label can collapse into one output column even when their internal names differ. Keep both names and labels unique when all metrics must remain visible.

Feature labels become outer row-index values. Reusing one display label for multiple registered features can create duplicate (feature label, level) index keys when level names overlap, making selection, export, and styling ambiguous. Keep feature labels unique in report-oriented workflows.

Ordinary and error evaluation normalize feature-level keys to strings. Non-categorical columns are converted to strings before grouping and normally appear in pandas’ default lexicographically sorted group order, while categorical group keys are stringified during result assembly. Distinct native values with the same string representation—for example integer 1 and string "1"—raise ValueError. Normalize level values to unambiguous labels before evaluation.

Reserved internal column names

Avoid using these names for ordinary audit-feature columns:

_truth
_pred
_binary_pred
overall
tp
tn
fp
fn

add_outcome() writes _truth into the auditor’s internal DataFrame copy. Evaluation then writes _pred, _binary_pred, the synthetic overall column, and requested confusion indicators into a temporary slice. add_feature() rejects those reserved names with ValueError. A pre-existing _truth column does not replace outcome registration: evaluation requires add_outcome() to have been called for the current data.

The caller’s original DataFrame is protected by add_data()’s copy. Rename conflicting feature columns and create truth through add_outcome().

HierarchyPlotter also uses _temp_feature inside its copied compile-time DataFrame when multiple hierarchy items are valid at one level. Treat that name as reserved within hierarchy definitions.

Data replacement

add_data() replaces the stored DataFrame copy without replaying earlier outcome setup. Configure the outcome after the final data replacement.

Features, scores, and selected metrics remain registered unless replaced explicitly, so confirm that their column names still match the new DataFrame.