Built-in metrics

Import built-in metrics from:

from model_auditor import metrics

Or import classes directly:

from model_auditor.metrics import (
    AUROC,
    Sensitivity,
    Specificity,
)

Metric protocol

An ordinary metric supplies:

class AuditorMetric(Protocol):
    name: str
    label: str
    inputs: list[str]
    ci_eligible: bool
    direction: str

    def data_call(
        self,
        data: pd.DataFrame,
    ) -> float | int:
        ...

name becomes the key in LevelEvaluation.metrics. label is the required human-readable display text. inputs controls which confusion indicators the auditor derives.

Classification metrics

Let TP, TN, FP, and FN be the summed confusion indicators for the current feature level.

Class	name	label	Inputs	Calculation	CI
Sensitivity	sensitivity	Sensitivity	tp, fn	TP / (TP + FN)	Yes
Specificity	specificity	Specificity	tn, fp	TN / (TN + FP)	Yes
Precision	precision	Precision	tp, fp	TP / (TP + FP)	Yes
Recall	recall	Recall	tp, fn	TP / (TP + FN)	Yes
F1Score	f1	F1 Score	tp, fp, fn	harmonic mean of precision and recall	Yes
FBetaScore	dynamic	dynamic	tp, fp, fn	weighted harmonic mean of precision and recall	Yes
MatthewsCorrelationCoefficient	mcc	Matthews Correlation Coefficient	tp, tn, fp, fn	(TP×TN - FP×FN) / sqrt((TP+FP)(TP+FN)(TN+FP)(TN+FN))	Yes
TPR	tpr	TPR	tp, fn	sensitivity alias	Yes
TNR	tnr	TNR	tn, fp	specificity alias	Yes
FPR	fpr	FPR	fp, tn	FP / (FP + TN)	Yes
FNR	fnr	FNR	fn, tp	FN / (FN + TP)	Yes
NegativePredictiveValue	npv	NPV	tn, fn	TN / (TN + FN)	Yes
BalancedAccuracy	balanced_accuracy	Balanced accuracy	tp, tn, fp, fn	mean of sensitivity and specificity	Yes
SelectionRate	selection_rate	Selection rate	_binary_pred	predicted-positive proportion	Yes
ExpectedCost	expected_cost	Expected cost	fp, fn	weighted FP/FN cost per row	Yes

Undefined ratios return NaN. MCC also returns NaN when its denominator is zero.

FBetaScore

FBetaScore(
    beta: float = 1.0,
)

Calculation:

(1 + beta²) × precision × recall
---------------------------------
beta² × precision + recall

Beta must be finite and positive. The instance name and label preserve the normalized float value:

Construction	Name	Label
FBetaScore(beta=0.5)	f0_5	F0.5 Score
FBetaScore(beta=1.0)	f1_0	F1.0 Score
FBetaScore(beta=2.0)	f2_0	F2.0 Score

Continuous-score metrics

Class	name	label	Inputs	Implementation	CI
AUROC	auroc	AUROC	_truth, _pred	scikit-learn roc_auc_score	Yes
AUPRC	auprc	AUPRC	_truth, _pred	scikit-learn average_precision_score	Yes
AveragePrecision	average_precision	Average precision	_truth, _pred	scikit-learn average_precision_score	Yes

_pred is the original continuous score, not _binary_pred. These metrics do not change solely because a decision threshold changes.

When the underlying scikit-learn call raises ValueError, the metric returns NaN.

AUPRC is retained as a compatibility name for non-interpolated average precision; it is not trapezoidal PR area.

Probability and descriptive metrics

BrierScore, LogLoss, CalibrationIntercept, and CalibrationSlope require finite probabilities in [0, 1]. Calibration fits also require both truth classes and identifiable, interior-probability data. Prevalence reports the observed positive proportion. See Evaluate calibration and decisions.

Count metrics

Class	name	label	Inputs	Result	CI
nData	n	N	none	number of rows in the level	No
nTP	n_tp	TP	tp	sum of TP indicators	No
nTN	n_tn	TN	tn	sum of TN indicators	No
nFP	n_fp	FP	fp	sum of FP indicators	No
nFN	n_fn	FN	fn	sum of FN indicators	No
nPositive	n_pos	Pos.	_truth	count where truth equals 1	No
nNegative	n_neg	Neg.	_truth	count where truth equals 0	No

Count metrics are excluded from ordinary tier styling unless include_count_metrics=True.

Metric inputs

model_auditor.metric_inputs defines the structural AuditorMetricInput protocol:

AuditorMetricInput
  name: str
  label: str
  inputs: list[str]
  row_call(row: pd.Series) -> int | float
  data_transform(data: pd.DataFrame) -> pd.DataFrame

The recognized built-in intermediate classes are:

Class	Output column	Row condition
TruePositives	tp	truth 1 and binary prediction 1
FalsePositives	fp	truth 0 and binary prediction 1
TrueNegatives	tn	truth 0 and binary prediction 0
FalseNegatives	fn	truth 1 and binary prediction 0

The vectorized data_transform() implementation produces integer indicator columns. Although the protocol is public, Auditor discovers calculators from the package module and exposes no third-party registration method in v0.1.16.

Recommended metric sets

Compact classification report

auditor.set_metrics(
    [
        Sensitivity(),
        Specificity(),
        Precision(),
        AUROC(),
        AUPRC(),
        nData(),
        nPositive(),
        nNegative(),
    ]
)

Full confusion report

auditor.set_metrics(
    [
        Sensitivity(),
        Specificity(),
        Precision(),
        F1Score(),
        MatthewsCorrelationCoefficient(),
        FPR(),
        FNR(),
        nData(),
        nTP(),
        nTN(),
        nFP(),
        nFN(),
        nPositive(),
        nNegative(),
    ]
)

Ranking-only values

auditor.set_metrics(
    [
        AUROC(),
        AUPRC(),
        nData(),
        nPositive(),
        nNegative(),
    ]
)

evaluate_metrics() requires a threshold only when at least one selected metric uses threshold-derived inputs. Ranking-only selections do not require one.

Custom metrics

A custom metric can use _truth, _pred, or the recognized confusion inputs. See Create a custom metric.

Metric names must be unique; evaluation rejects duplicates. Keep display labels unique as well when using metric_labels=True.