Diagnostic accuracy and classifier metrics

A binary classifier, such as a diagnostic test or a risk score with a cutoff, sorts units into alerts and non-alerts. Compared with the true class, this gives four counts: true positives (TP), false negatives (FN), false positives (FP) and true negatives (TN). infercalcR calculates the usual accuracy metrics from these counts, keeps track of their denominators and reports an explicit status when a metric is undefined. With scored observations it also evaluates cutoffs, ROC summaries and cutoff selection.

All data in this tutorial are synthetic. Calculations from counts need only base R. Specifications built from scored observations record a canonical fingerprint of their inputs and therefore use the optional packages digest and jsonlite; the chunks that need them are evaluated only when both are installed.

has_fingerprints <- requireNamespace("digest", quietly = TRUE) &&
  requireNamespace("jsonlite", quietly = TRUE)
has_fingerprints
#> [1] TRUE

Metrics from four counts

counts <- c(tp = 40, fn = 10, fp = 20, tn = 130)
metrics <- classification_metrics(counts)
metrics[metrics$metric_id %in% c("prevalence", "tpr", "tnr", "ppv", "npv",
                                 "lr_pos", "lr_neg", "youden_j"),
        c("metric_id", "value", "status", "denominator", "denominator_value")]
#>     metric_id     value status
#> 10 prevalence 0.2500000     ok
#> 12        tpr 0.8000000     ok
#> 14        tnr 0.8666667     ok
#> 16        ppv 0.6666667     ok
#> 18        npv 0.9285714     ok
#> 22   youden_j 0.6666667     ok
#> 31     lr_pos 6.0000000     ok
#> 32     lr_neg 0.2307692     ok
#>                                                  denominator denominator_value
#> 10                                             N (all units)       200.0000000
#> 12                                   TP + FN (actual events)        50.0000000
#> 14                               FP + TN (actual non-events)       150.0000000
#> 16                                          TP + FP (alerts)        60.0000000
#> 18                               FN + TN (units not alerted)       140.0000000
#> 22 TP + FN and FP + TN (actual events and actual non-events)                NA
#> 31                                 FPR (false-positive rate)         0.1333333
#> 32                                  TNR (true-negative rate)         0.8666667

Sensitivity (tpr) divides TP by all events and specificity (tnr) divides TN by all non-events. The positive predictive value (ppv) divides TP by all alerts, and the negative predictive value (npv) divides TN by all non-alerts. The likelihood ratios combine sensitivity and specificity, and Youden’s J is their sum minus one. The denominator column states which counts each proportion uses.

When a denominator is zero, the metric is not set to zero or one. Its value is NA, its status is "undefined" and the reason is given.

no_alerts <- classification_metrics(c(tp = 0, fn = 5, fp = 0, tn = 95))
no_alerts[no_alerts$metric_id %in% c("tpr", "ppv", "npv", "lr_pos"),
          c("metric_id", "value", "status", "reason")]
#>    metric_id value    status                  reason
#> 12       tpr  0.00        ok                    <NA>
#> 16       ppv    NA undefined no alerts (TP + FP = 0)
#> 18       npv  0.95        ok                    <NA>
#> 31    lr_pos    NA undefined     TPR = FPR = 0 (0/0)

metric_dictionary() documents every metric, including its formula and whether it depends on prevalence.

dictionary <- metric_dictionary()
dictionary[dictionary$id %in% c("tpr", "ppv", "lr_pos"),
           c("id", "name", "formula", "prevalence_dependent")]
#>        id                                     name        formula
#> 12    tpr True-positive rate (sensitivity, recall) TP / (TP + FN)
#> 16    ppv    Positive predictive value (precision) TP / (TP + FP)
#> 31 lr_pos                Positive likelihood ratio      TPR / FPR
#>    prevalence_dependent
#> 12                FALSE
#> 16                 TRUE
#> 31                FALSE

Prevalence and predictive values

Sensitivity and specificity describe the test within each class. Predictive values also depend on how common events are. If sensitivity and specificity carry over unchanged to another setting, Bayes’ theorem gives the predictive values at an assumed prevalence. These are scenario values, not new observations.

tpr <- 40 / 50
fpr <- 20 / 150
predictive_values_at_prevalence(tpr, fpr, prevalence = c(0.01, 0.05, 0.25))
#>   prevalence        ppv ppv_status ppv_reason_code       npv npv_status
#> 1       0.01 0.05714286         ok            <NA> 0.9976744         ok
#> 2       0.05 0.24000000         ok            <NA> 0.9880000         ok
#> 3       0.25 0.66666667         ok            <NA> 0.9285714         ok
#>   npv_reason_code
#> 1            <NA>
#> 2            <NA>
#> 3            <NA>
prevalence_for_ppv_target(tpr, fpr, target = 0.5)
#>   tpr       fpr target prevalence_needed status reason_code reason
#> 1 0.8 0.1333333    0.5         0.1428571     ok        <NA>   <NA>

At a prevalence of 1 %, only about 6 % of alerts are events, although the test is the same. A prevalence of about 14 % is needed for half of the alerts to be events.

Intervals for a fixed-cutoff table

The diagnostic_2x2_fixed_cutoff method works through the common interface described in the getting-started tutorial. It requires the sampling design and the origin of the cutoff to be declared, and reports Clopper-Pearson intervals that are conditional on the fixed rule.

dx_spec <- method_spec("diagnostic_2x2_fixed_cutoff", conf.level = 0.95)
dx <- run_inference(dx_spec, tp = 40, fn = 10, fp = 20, tn = 130,
                    sampling = "cross_sectional",
                    cutoff_origin = "predeclared_rule",
                    evaluation_split = "held_out_test")
dx_rows <- result_data(dx)
print(dx_rows[dx_rows$estimand %in% c("tpr", "tnr", "ppv", "npv"),
              c("estimand", "estimate", "conf.low", "conf.high",
                "interval.method", "status")],
      row.names = FALSE)
#>  estimand  estimate  conf.low conf.high interval.method status
#>       tpr 0.8000000 0.6628169 0.8996978 clopper_pearson     ok
#>       tnr 0.8666667 0.8016130 0.9166162 clopper_pearson     ok
#>       ppv 0.6666667 0.5331273 0.7831306 clopper_pearson     ok
#>       npv 0.9285714 0.8725637 0.9652176 clopper_pearson     ok

Four counts identify a single operating point. They cannot reproduce a ROC curve, an AUC or a sweep over cutoffs; those need the scores.

Scored observations

The next example simulates a continuous marker. Events have a mean marker value 1.2 standard deviations above non-events, and about 30 % of units are events. Two independent samples are drawn: one to tune a cutoff and one to evaluate it.

simulate_marker <- function(n, first_id) {
  event <- rbinom(n, size = 1, prob = 0.3)
  data.frame(id = first_id + seq_len(n) - 1L,
             event = event,
             marker = round(rnorm(n, mean = 1.2 * event, sd = 1), 2))
}
set.seed(2026)
tuning <- simulate_marker(200, first_id = 1L)
evaluation <- simulate_marker(200, first_id = 1001L)
c(tuning = mean(tuning$event), evaluation = mean(evaluation$event))
#>     tuning evaluation 
#>       0.24       0.29

A specification names the truth column, the positive class, the score column, the score direction and the comparison at the cutoff. None of these are guessed. It also records where the data come from relative to the cutoff and how the units were sampled. Here the cutoff 0.5 is fixed in advance.

marker_spec <- function(data, split) {
  classification_spec(
    data = data, truth = "event", positive_label = 1, negative_label = 0,
    scores = "marker", id = "id", cutoff = 0.5,
    direction = "higher_is_positive", comparator = ">=",
    evaluation_split = split, sampling = "cross_sectional"
  )
}
tuning_spec <- marker_spec(tuning, split = "same_data")
fixed <- evaluate_cutoff(tuning_spec)
fixed
#> <classification_result> scored_data
#>   status:     ok
#>   evaluation: A manual cutoff evaluated on data that may have informed it; treat the values as descriptive.
#>   model marker (ok)
#>     rule:   alert if score >= 0.5
#>     counts: TP 41, FN 7, FP 51, TN 101 (observed)
#>     TPR 0.8542; FPR 0.3355; PPV 0.4457; NPV 0.9352
#>     ROC AUC: 0.8465
#>   limitation: same_data_manual_cutoff
#>   limitation: apparent_performance
#>   limitation: fixed_cutoff_intervals
#>   limitation: post_selection_not_computed
#>   spec hash:  322f6106db836e1b2e958a78140a72e0df78b517904fbc1d154f7ebf4d85fa61

The printed result lists the counts, the main metrics, the ROC AUC and the limitations that apply to this evaluation. classification_table() returns the metrics and their intervals as data frames.

intervals <- classification_table(fixed, "intervals")
intervals[intervals$metric_id %in% c("tpr", "fpr"),
          c("metric_id", "method", "level", "lower", "upper")]
#>    metric_id          method level     lower     upper
#> 5        tpr          wilson  0.95 0.7283282 0.9275175
#> 6        tpr clopper_pearson  0.95 0.7223617 0.9392955
#> 11       fpr          wilson  0.95 0.2653373 0.4138239
#> 12       fpr clopper_pearson  0.95 0.2610691 0.4165382
classification_auc(tuning_spec)[, c("metric_id", "value", "ci_lower",
                                    "ci_upper", "setting")]
#>   metric_id     value  ci_lower  ci_upper                      setting
#> 1   roc_auc 0.8464912 0.7866625 0.9063199 Mann-Whitney, ties count 1/2

cutoff_sweep() evaluates every distinct decision set of the observed scores, from alerting no unit to alerting all units. Its rates trace the empirical ROC curve.

sweep <- cutoff_sweep(tuning_spec)
nrow(sweep)
#> [1] 159
head(sweep[, c("cutoff", "alerts", "tp", "fp", "tpr", "fpr")])
#>   cutoff alerts tp fp        tpr         fpr
#> 1     NA      0  0  0 0.00000000 0.000000000
#> 2   3.38      1  1  0 0.02083333 0.000000000
#> 3   3.36      2  2  0 0.04166667 0.000000000
#> 4   2.73      3  3  0 0.06250000 0.000000000
#> 5   2.68      4  4  0 0.08333333 0.000000000
#> 6   2.64      5  4  1 0.08333333 0.006578947
plot(sweep$fpr, sweep$tpr, type = "l", xlim = c(0, 1), ylim = c(0, 1),
     xlab = "False positive rate", ylab = "True positive rate",
     main = "Tuning sample")
abline(0, 1, lty = 3)
point <- fixed$models[[1]]$metrics
points(point$value[point$metric_id == "fpr"],
       point$value[point$metric_id == "tpr"], pch = 19)

Empirical ROC curve of the simulated marker on the tuning sample, with the point for the prespecified cutoff 0.5 marked.

Select a cutoff, then evaluate it on new data

select_cutoff() searches all distinct decision sets for the cutoff that optimizes a declared objective. Here the objective is Youden’s J.

selected <- select_cutoff(tuning_spec, "max_youden")
selected$models[[1]]$operating_point$cutoff
#> [1] 0.63
classification_table(selected, "selection")[, c("procedure", "objective_value",
                                                "n_candidates", "n_tied",
                                                "tie_break")]
#>    procedure objective_value n_candidates n_tied     tie_break
#> 1 max_youden       0.5559211          159      1 fewest_alerts

A cutoff chosen on the same data that are used to describe it gives in-sample values. To evaluate it, pass the tuning result as the cutoff for a specification of the independent sample.

evaluation_spec <- marker_spec(evaluation, split = "held_out_test")
held_out <- evaluate_cutoff(evaluation_spec, cutoff = selected)
held_out$evaluation$label
#> [1] "held_out_rule"
compare <- function(result, ids = c("tpr", "fpr", "youden_j")) {
  m <- result$models[[1]]$metrics
  setNames(m$value[match(ids, m$metric_id)], ids)
}
rbind(in_sample = compare(selected), held_out = compare(held_out))
#>                 tpr       fpr  youden_j
#> in_sample 0.8125000 0.2565789 0.5559211
#> held_out  0.7068966 0.3450704 0.3618261

Because the data were simulated, the true operating characteristics are known. binormal_rates() computes them for a binormal model with a given AUC, with the cutoff on the scale of the non-event distribution. With a mean difference of 1.2 standard deviations and equal variances, the AUC is pnorm(1.2 / sqrt(2)) and Youden’s J is largest at the midpoint 0.6.

true_auc <- pnorm(1.2 / sqrt(2))
truth <- binormal_rates(cutoff = c(0.5, 0.6), auc = true_auc)
truth$youden_j <- truth$tpr - truth$fpr
truth
#>   cutoff       tpr       fpr  youden_j
#> 1    0.5 0.7580363 0.3085375 0.4494988
#> 2    0.6 0.7257469 0.2742531 0.4514938

In this simulated split, the in-sample Youden’s J of the selected cutoff is higher than the true value of about 0.45 and the held-out value is lower. One split illustrates the variability of these estimates; it does not quantify the optimism of in-sample selection.

Common result contract

as_inference() converts a classification result to the common result contract. The returned object keeps aggregate values and intervals but not the individual observations or identifiers.

common <- as_inference(held_out)
rows <- result_data(common)
print(rows[rows$estimand %in% c("tpr", "fpr", "ppv", "roc_auc"),
           c("estimand", "estimate", "conf.low", "conf.high",
             "interval.method")],
      row.names = FALSE)
#>  estimand  estimate  conf.low conf.high interval.method
#>       tpr 0.7068966 0.5798710 0.8082182          wilson
#>       tpr 0.7068966 0.5727030 0.8190930 clopper_pearson
#>       fpr 0.3450704 0.2718894 0.4264132          wilson
#>       fpr 0.3450704 0.2673621 0.4293928 clopper_pearson
#>       ppv 0.4555556 0.3565962 0.5581537          wilson
#>       ppv 0.4555556 0.3501547 0.5639634 clopper_pearson
#>   roc_auc 0.7795653 0.7084650 0.8506657          delong
result_metadata(common)$evaluation$evaluation_split
#> [1] "held_out_test"

Notes on interpretation

The positive class, score direction and comparator are declarations. The package does not infer them from the data.
The sampling and evaluation_split arguments document the design; they do not verify it. Independent tuning and evaluation samples must be ensured by the study design.
Proportion intervals are conditional on the cutoff. They do not account for the selection of that cutoff or for clustered observations.
In a case-control sample, the share of events is fixed by design, so predictive values from those counts do not estimate the population values.

The exact tests tutorial covers comparisons of two independent proportions.

Acknowledgments and sources

This package is part of an inference suite (infercalcR, inferguidR and infereportR) that is deeply inspired by DescrTab2 [1], developed at the Institute of Medical Biometry (IMBI), Heidelberg University, by Jan Meis, Lukas Baumann, Maximilian Pilz and Lukas Sauer. Its test-choice tree, its validation statement and its practice of computing each test with an established implementation are the model for how the suite chooses, computes and reports statistical tests. DescrTab2 was archived on CRAN on 18 October 2025; its source code remains available on GitHub. The methods-consulting pages of the University of Zurich [2], with their decision tree, decision assistant, worked examples and explanations, inspired the guidance in the suite. Power and sample-size planning and the distribution calculator follow G*Power 3.1 [3, 4]. We thank the authors of these works for making them openly available. The package README cites all software the package uses.

Meis J, Baumann L, Pilz M, Sauer L (2022). DescrTab2: Publication Quality Descriptive Statistics Tables. R package version 2.1.16. https://doi.org/10.32614/CRAN.package.DescrTab2. Source code, including the development version 2.1.26: https://github.com/imbi-heidelberg/DescrTab2.
Universität Zürich, Methodenberatung. Datenanalyse mit SPSS: decision tree (Entscheidbaum), decision assistant (Entscheidassistent) and method pages with examples and explanations. https://www.methodenberatung.uzh.ch/de/datenanalyse_spss.html. Accessed 3 October 2026.
Faul F, Erdfelder E, Lang A-G, Buchner A (2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39(2), 175-191. https://doi.org/10.3758/BF03193146.
Faul F, Erdfelder E, Buchner A, Lang A-G (2009). Statistical power analyses using G*Power 3.1: Tests for correlation and regression analyses. Behavior Research Methods, 41(4), 1149-1160. https://doi.org/10.3758/BRM.41.4.1149.

sessionInfo()
#> R version 4.6.2 (2026-10-29)
#> Platform: x86_64-pc-linux-gnu
#> Running under: Ubuntu 26.04.1 LTS
#> 
#> Matrix products: default
#> BLAS:   /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3 
#> LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.32.so;  LAPACK version 3.12.0
#> 
#> locale:
#>  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
#>  [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
#>  [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
#>  [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
#>  [9] LC_ADDRESS=C               LC_TELEPHONE=C            
#> [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       
#> 
#> time zone: Etc/UTC
#> tzcode source: system (glibc)
#> 
#> attached base packages:
#> [1] stats     graphics  grDevices utils     datasets  methods   base     
#> 
#> other attached packages:
#> [1] infercalcR_0.1.0 rmarkdown_2.32  
#> 
#> loaded via a namespace (and not attached):
#>  [1] digest_0.6.39    R6_2.6.1         fastmap_1.2.0    xfun_0.62       
#>  [5] maketools_1.3.2  cachem_1.1.0     knitr_1.52       htmltools_0.5.9 
#>  [9] buildtools_1.0.0 lifecycle_1.0.5  cli_3.6.6        sass_0.4.10     
#> [13] jquerylib_0.1.4  compiler_4.6.2   sys_3.4.3        tools_4.6.2     
#> [17] evaluate_1.0.5   bslib_0.12.0     yaml_2.3.12      otel_0.3.0      
#> [21] jsonlite_2.0.0   rlang_1.3.0