Testing and verification

Run the complete suite

From an editable source checkout:

python -m pytest

Matplotlib must be installed because plotting tests import it and use the non-interactive Agg backend.

Run targeted modules

Metric formulas and inputs

python -m pytest \
  tests/test_metrics_correctness.py \
  tests/test_metric_inputs_correctness.py

These tests compare formulas with known values and scikit-learn oracles, including zero-denominator behavior.

Ordinary evaluation and thresholds

python -m pytest \
  tests/test_evaluate_metrics_correctness.py \
  tests/test_threshold_optimization.py

Coverage includes:

• overall and subgroup known values;
• scalar threshold overrides;
• conditional thresholds;
• target tie breaking;
• infeasible target errors; and
• Youden threshold selection on deterministic fixtures.

Bootstrap and categorical behavior

python -m pytest \
  tests/test_bootstrap_confidence_intervals.py \
  tests/test_feature_level_ordering.py

Coverage includes seeded bootstrap oracles, CI eligibility, declared category order, and unobserved placeholders.

Error analysis

python -m pytest tests/test_error_evaluation.py

Coverage includes:

• TP/TN/FP/FN structure;
• exact odds ratios;
• zero, infinity, and NaN cases;
• support counts;
• wide DataFrame output;
• conditional thresholds; and
• bootstrap intervals.

Tables and plots

python -m pytest \
  tests/test_result_styling.py \
  tests/test_score_evaluation_plotting.py \
  tests/test_auditor_score_distribution_plotting.py

Coverage includes tier direction, count exclusion, metric-name/label resolution, annotations, overall comparators, shared bins, density normalization, and level order.

Add deterministic tests

For a metric or threshold change, build a small table where TP, TN, FP, and FN can be counted by inspection.

import pandas as pd

data = pd.DataFrame(
    {
        "group": ["A", "A", "A", "B", "B", "B"],
        "score": [0.9, 0.7, 0.2, 0.8, 0.4, 0.1],
        "label": [1, 0, 0, 1, 1, 0],
    }
)

Assert exact counts first, then ratios.

For resampling:

from model_auditor import InferenceConfig

inference = InferenceConfig(random_state=12345)

Use explicit local seeds for both the independent oracle and implementation call; tests should also confirm the global NumPy state is unchanged.

Test public edge behavior

A behavioral change should consider:

• no data, outcome, score, metric, or threshold;
• unknown score and feature names;
• score exactly equal to threshold;
• null feature, score, and outcome values;
• single-class feature levels;
• categorical unobserved levels;
• incomplete conditional mappings;
• finite validation for every threshold;
• denominator-zero metrics;
• zero, NaN, and infinite odds ratios;
• positive and disabled bootstrap paths;
• metric names versus labels;
• display-formatted versus numeric exports; and
• missing optional plotting dependencies.

Build after tests

python -m build

Inspect both source and wheel artifacts before release when package inclusion changes.

Current automation boundary

The stable release workflow calls the shared validation workflow before publishing. It runs pytest on Python 3.10, 3.11, and 3.12 plus Ruff lint, formatting, and package-build checks.