Thresholds and predictions

A continuous score becomes a binary prediction only after Model Auditor resolves an operating threshold for each row.

Comparison rule

The stable release predicts positive when:

score >= threshold

Equality belongs to the positive class. This convention affects observations exactly at an ROC threshold and should match any external confusion-matrix calculation used for verification.

Threshold precedence

For evaluate_metrics() and evaluate_errors():

1. a threshold passed to the method wins;
2. otherwise the registered AuditorScore.threshold is used;
3. if neither exists and a selected metric needs binary predictions, evaluation raises ValueError.

A call-time scalar replaces a stored conditional policy, and a call-time conditional policy replaces a stored scalar.

Scalar thresholds

A scalar is converted to float and must be finite.

auditor.add_score(
    name="risk_score",
    threshold=0.5,
)

Invalid examples include:

NaN
+Infinity
-Infinity
"not numeric"

Numeric strings can be converted by float, although using actual numeric values is clearer.

Conditional thresholds

ConditionalThreshold selects a value from another column:

ConditionalThreshold(
    feature="region",
    levels={
        "North": 0.25,
        "South": 0.45,
    },
    default=0.50,
)

Resolution uses pandas mapping over the condition column’s native values. Stringifying a mapping key can therefore break a match when the source values are integers, timestamps, or another type.

A condition feature is included in the internal evaluation slice even when it is not registered as an audit feature.

Complete mapping without default

When default is absent:

• every observed non-null level must appear in levels;
• a null condition value is an error.

Fallback mapping with default

When default is present:

• an unmapped level uses it;
• a null condition value uses it.

Every level and default threshold is validated as finite.

Scalar optimization

The two built-in threshold optimizers operate on the registered score and _truth values.

Youden index

optimize_score_threshold() chooses the first ROC threshold maximizing:

TPR - FPR

The method considers finite observed score values and returns the first threshold maximizing the criterion. Both truth classes are required.

Target sensitivity or specificity

optimize_score_threshold_for_target() builds the full ROC threshold list with intermediate points retained.

For a minimum sensitivity, it returns the highest feasible finite threshold.

For a minimum specificity, it returns the lowest feasible finite threshold. The candidate set includes a finite threshold just above the maximum observed score when floating-point representation permits it, so an all-negative operating point can be selected.

Both methods return a value and emit a warning; they do not mutate score configuration.

Ranking metrics do not depend on the threshold

AUROC and AUPRC read _truth and the continuous _pred column. Changing only the threshold should not change those values.

The following depend on binary predictions:

• sensitivity and recall;
• specificity;
• precision;
• F1 and F-beta;
• MCC;
• TPR, TNR, FPR, and FNR;
• TP, TN, FP, and FN counts; and
• error-group odds ratios.

Total N and truth-class counts do not intrinsically depend on the threshold, although they are evaluated in the same threshold-resolved pipeline.

Selection bias and deployment policy

A numerically optimal threshold is not automatically the correct deployment policy. Consider:

• separate data for threshold selection and final estimation;
• asymmetric costs;
• calibration and prevalence shift;
• minimum support around candidate operating points;
• subgroup policy constraints;
• monitoring for new conditional levels; and
• documentation of the exact threshold used in every report.

Model Auditor supplies deterministic threshold rules; it does not estimate causal or policy legitimacy.