Optimize a scalar threshold

This guide selects one scalar threshold from a labeled evaluation dataset and then applies it explicitly.

Prerequisites

Register data, a score, and an outcome. The score does not need an existing threshold for optimization.

from model_auditor import Auditor

auditor = Auditor()
auditor.add_data(selection_data)
auditor.add_score(name="risk_score", label="Risk score")
auditor.add_outcome(name="outcome")

Optimization requires both outcome classes and numeric scores that scikit-learn can use to build an ROC curve.

Separate selection from final evaluation

Choosing a threshold and reporting final performance on the same observations can produce optimistic estimates. Select the operating point on training or validation data, then evaluate the fixed threshold on an independent test set when unbiased reporting matters.

Balance sensitivity and specificity with Youden index

optimize_score_threshold() maximizes:

sensitivity - false positive rate

This is equivalent to maximizing sensitivity + specificity - 1.

import warnings

with warnings.catch_warnings():
    warnings.simplefilter("ignore", UserWarning)
    youden_threshold = auditor.optimize_score_threshold(
        score_name="risk_score",
    )

print(youden_threshold)

The method returns a float and emits a UserWarning containing the selected value. It does not update the registered score.

The Youden method considers finite observed score thresholds and returns the first threshold maximizing the criterion. It rejects data without both truth classes.

import numpy as np

assert np.isfinite(youden_threshold)

Apply the returned value explicitly:

auditor.scores["risk_score"].threshold = youden_threshold

A less stateful alternative is to pass the value to each evaluation call:

results = auditor.evaluate_metrics(
    score_name="risk_score",
    threshold=youden_threshold,
    n_bootstraps=None,
)

Require minimum sensitivity

The target optimizer chooses the highest finite threshold whose ROC point satisfies the requested sensitivity. This preserves the most restrictive threshold among feasible choices.

sensitivity_threshold = auditor.optimize_score_threshold_for_target(
    score_name="risk_score",
    target=0.90,
    metric="sensitivity",
)

Verify the constraint when evaluating the same selection data:

from model_auditor.metrics import Sensitivity

auditor.set_metrics([Sensitivity()])

check = auditor.evaluate_metrics(
    score_name="risk_score",
    threshold=sensitivity_threshold,
    n_bootstraps=None,
)

observed = (
    check.features["overall"]
    .levels["Overall"]
    .metrics["sensitivity"]
    .score
)

assert observed >= 0.90

Require minimum specificity

For specificity, the method chooses the lowest finite threshold among ROC points that satisfy the target.

specificity_threshold = auditor.optimize_score_threshold_for_target(
    score_name="risk_score",
    target=0.95,
    metric="specificity",
)

The tie-breaking rules are part of v0.1.16 behavior and may matter when several observations share a score.

Handle an infeasible target

The target must be between 0.0 and 1.0, inclusive, and the metric must be exactly "sensitivity" or "specificity".

When no finite ROC threshold satisfies the request, the method raises ValueError and reports the achievable range across finite thresholds.

try:
    threshold = auditor.optimize_score_threshold_for_target(
        score_name="risk_score",
        target=0.999,
        metric="specificity",
    )
except ValueError as exc:
    print(exc)

Do not silently replace an infeasible target with infinity or another sentinel. Revisit the requirement, score model, or selection population.

What optimization does not do

• It does not create subgroup-specific thresholds.
• It does not update an AuditorScore.
• It does not bootstrap threshold uncertainty.
• It does not optimize precision, F1, cost, utility, or calibration.
• It does not protect against selection bias or dataset shift.

Use conditional thresholds when deployment policy supplies different thresholds by feature level.