Create a custom metric

Model Auditor uses structural typing for metrics. A custom metric needs the expected attributes and a data_call() method; it does not need registration or inheritance at runtime.

Choose supported inputs

The stable evaluator recognizes these metric inputs:

Input	Meaning
_truth	Binary ground truth created by add_outcome()
_pred	Continuous score being evaluated
tp	Row-level true-positive indicator
tn	Row-level true-negative indicator
fp	Row-level false-positive indicator
fn	Row-level false-negative indicator

List every input your metric needs. The auditor derives only the requested confusion columns.

Arbitrary third-party names cannot be listed in inputs because v0.1.16 discovers calculators only from the package’s built-in metric-input module. The DataFrame passed to data_call() also contains registered audit features and, when applicable, the conditional-threshold feature, but those columns are context-dependent rather than portable metric inputs. A reusable custom metric should normally use the recognized names above; add a package-level input calculator when another guaranteed input is required.

Implement accuracy

import pandas as pd

class Accuracy:
    name = "accuracy"
    label = "Accuracy"
    inputs = ["tp", "tn", "fp", "fn"]
    ci_eligible = True

    def data_call(self, data: pd.DataFrame) -> float:
        tp = int(data["tp"].sum())
        tn = int(data["tn"].sum())
        fp = int(data["fp"].sum())
        fn = int(data["fn"].sum())

        denominator = tp + tn + fp + fn
        if denominator == 0:
            return 0.0

        return (tp + tn) / denominator

Use it alongside built-in metrics:

from model_auditor.metrics import AUROC, Sensitivity

auditor.set_metrics(
    [
        Accuracy(),
        Sensitivity(),
        AUROC(),
    ]
)

results = auditor.evaluate_metrics(
    score_name="risk_score",
    n_bootstraps=1000,
)

With ci_eligible=True, the metric runs on every bootstrap sample. Use False for counts or values for which percentile intervals are not meaningful.

Implement a score-only metric

A metric can use the continuous score without requiring a threshold-derived input:

class MeanScore:
    name = "mean_score"
    label = "Mean score"
    inputs = ["_pred"]
    ci_eligible = True

    def data_call(self, data: pd.DataFrame) -> float:
        return float(data["_pred"].mean())

The evaluation method requires a threshold only when a selected metric declares a threshold-derived input such as _binary_pred, tp, tn, fp, or fn.

Implement a truth-prevalence metric

class Prevalence:
    name = "prevalence"
    label = "Prevalence"
    inputs = ["_truth"]
    ci_eligible = True

    def data_call(self, data: pd.DataFrame) -> float:
        return float(data["_truth"].mean())

Validate truth values before relying on this calculation.

Follow metric identity rules

• name should be stable, machine-readable, and unique within the selected metric list.
• label should be concise and human-readable.
• Return one Python or NumPy scalar, not a Series or DataFrame.
• Define denominator-zero and missing-data behavior explicitly.
• Avoid mutating the supplied DataFrame.
• Keep the result meaningful within a single feature level.

Duplicate names overwrite the same result key during assembly.

Understand bootstrap cost

A custom CI-eligible metric is called once for the observed level and once per bootstrap sample for every observed level. Avoid expensive model inference, disk access, or network calls inside data_call().

Styling custom metrics

Notebook styling is neutral unless rank=True. Set a custom metric’s direction to "higher", "lower", or "none" to control or suppress relative ranking.

Test known values

import pandas as pd

test_data = pd.DataFrame(
    {
        "tp": [1, 1, 0, 0],
        "tn": [0, 0, 1, 0],
        "fp": [0, 0, 0, 1],
        "fn": [0, 0, 0, 0],
    }
)

assert Accuracy().data_call(test_data) == 0.75

Add cases for zero denominators, nulls, single-class levels, and bootstrap compatibility before using a metric in reporting.

See Built-in metrics for the stable protocol and naming patterns.