Apply conditional thresholds

A ConditionalThreshold lets a single score use different decision thresholds for regions, cohorts, channels, or other discrete policy variables.

Define the threshold policy

from model_auditor import Auditor
from model_auditor.schemas import ConditionalThreshold

auditor = Auditor()
auditor.add_data(data)

auditor.add_feature(name="age_group", label="Age group")
auditor.add_score(
    name="risk_score",
    label="Risk score",
    threshold=ConditionalThreshold(
        feature="region",
        levels={
            "North": 0.25,
            "South": 0.45,
            "West": 0.70,
        },
        default=0.50,
    ),
)
auditor.add_outcome(name="outcome")

For each row, Model Auditor maps data["region"] to a threshold and predicts positive when:

risk_score >= resolved row threshold

The condition feature does not need to be registered with add_feature(). Register it only when you also want region-level result rows.

Decide whether to use a default

With default=0.50:

• feature values absent from levels use 0.50; and
• null feature values also use 0.50.

Without a default:

• every observed non-null feature value must have a mapping; and
• the condition feature must contain no null values.

An incomplete mapping raises a descriptive ValueError naming unresolved levels.

Defaults can hide policy drift

A default is operationally convenient, but a newly introduced feature level can begin using it silently. Monitor source levels before evaluation when every cohort requires explicit approval.

Evaluate metrics or errors

The same threshold policy applies to both public evaluation methods:

metric_results = auditor.evaluate_metrics(
    score_name="risk_score",
    n_bootstraps=None,
)

error_results = auditor.evaluate_errors(
    score_name="risk_score",
    n_bootstraps=None,
)

The result object does not add a per-row threshold column to the original DataFrame. add_data() stored a copy, and evaluation works from an internal slice.

Override the stored policy for one call

A call-time threshold replaces the entire stored threshold specification.

Use one scalar for all rows:

override_results = auditor.evaluate_metrics(
    score_name="risk_score",
    threshold=0.55,
    n_bootstraps=None,
)

Or supply a different conditional policy:

review_policy = ConditionalThreshold(
    feature="review_channel",
    levels={
        "automated": 0.70,
        "manual": 0.40,
    },
    default=0.50,
)

review_results = auditor.evaluate_metrics(
    score_name="risk_score",
    threshold=review_policy,
    n_bootstraps=None,
)

Validate the policy before evaluation

required_columns = {"risk_score", "outcome", "region"}
missing_columns = required_columns.difference(data.columns)
if missing_columns:
    raise ValueError(f"Missing required columns: {sorted(missing_columns)}")

unmapped = set(data["region"].dropna().unique()).difference(
    {"North", "South", "West"}
)
print(f"Levels using the default: {sorted(unmapped)}")
print(f"Null condition rows using the default: {data['region'].isna().sum()}")

Also check that every threshold is finite. NaN and positive or negative infinity are rejected.

Limitations

• The built-in optimizers return only scalar thresholds; they do not estimate a conditional mapping.
• HierarchyPlotter.set_score() accepts a threshold argument but does not use it in v0.1.16.
• A conditional policy can change subgroup confusion metrics in ways that make raw cross-group comparisons harder to interpret. Report the policy alongside the results.
• The package applies the policy you provide; it does not determine whether using a feature for thresholding is legally, ethically, or scientifically appropriate.

See Thresholds and predictions for the complete precedence and validation model.