Classification metrics and explicit operating points

This tutorial evaluates one operating point from synthetic counts and then from scored observations. It runs offline and does not train a model. Define the positive class, score direction, comparator and evaluation sample before interpreting performance.

Start with the four counts

spec <- classification_spec(
  counts = c(tp = 40, fn = 10, fp = 20, tn = 130), cutoff = 0.5
)
result <- evaluate_cutoff(spec)
metrics <- classification_metrics(result)
metrics[metrics$metric_id %in% c("tpr", "tnr", "ppv", "npv"),
        c("metric_id", "value", "status", "denominator_value")]
#>    metric_id     value status denominator_value
#> 12       tpr 0.8000000     ok                50
#> 14       tnr 0.8666667     ok               150
#> 16       ppv 0.6666667     ok                60
#> 18       npv 0.9285714     ok               140

Sensitivity (tpr) divides true positives by all events; specificity (tnr) divides true negatives by all non-events. Positive predictive value (ppv) divides true positives by all positive calls. These denominators differ. A missing denominator produces an explicit undefined result, not a zero. Aggregate counts identify one operating point; they cannot reconstruct a ROC curve or a cutoff sweep.

State how scored observations are evaluated

The following small data frame is entirely synthetic. The scores are declared as probabilities of the positive class; the comparator includes observations whose score is exactly 0.5. These are demonstration observations, not a held-out validation set.

d <- data.frame(
  id = seq_len(8),
  event = c(1, 1, 1, 0, 0, 0, 0, 0),
  risk = c(0.9, 0.8, 0.4, 0.7, 0.5, 0.2, 0.2, 0.1)
)
scored <- classification_spec(
  data = d, truth = "event", positive_label = 1, negative_label = 0,
  scores = "risk", id = "id", cutoff = 0.5,
  direction = "higher_is_positive", comparator = ">=",
  score_scale = "probability", evaluation_split = "same_data",
  sampling = "unknown"
)
evaluated <- evaluate_cutoff(scored)
tab <- classification_metrics(evaluated)
tab[tab$metric_id %in% c("tp", "fn", "fp", "tn", "tpr", "ppv"),
    c("metric_id", "value", "status")]
#>    metric_id     value status
#> 1         tp 2.0000000     ok
#> 2         fn 1.0000000     ok
#> 3         fp 2.0000000     ok
#> 4         tn 3.0000000     ok
#> 12       tpr 0.6666667     ok
#> 16       ppv 0.5000000     ok

A score’s numeric range alone does not establish probability calibration. The score_scale and sampling settings document assumptions; they do not validate them. Predictive values depend on prevalence, and the event proportion in an outcome-selected sample need not equal population prevalence. A prevalence scenario is a separate modelling assumption, not an observed population rate.

Inspect probability diagnostics at explicit thresholds

Calibration compares predicted probabilities with observed event frequencies. Net benefit uses a declared decision probability and its relative harm weight p / (1 - p). These displays require positive-class probabilities with higher_is_positive; neither is a prevalence-scenario calculation.

probability_result <- evaluate_cutoff(
  scored, calibration_bins = 3, threshold_probability = c(0.2, 0.5, 0.8)
)
classification_plot_data(probability_result, "calibration")$data$points
#>   group n_units events mean_predicted observed_rate min_predicted max_predicted
#> 1     1       3      0      0.1666667     0.0000000           0.1           0.2
#> 2     2       2      1      0.4500000     0.5000000           0.4           0.5
#> 3     3       3      2      0.8000000     0.6666667           0.7           0.9
classification_plot_data(probability_result, "net_benefit")$data$points
#>   threshold_probability harm_weight tp fp n_total net_benefit
#> 1                   0.2        0.25  3  4       8        0.25
#> 2                   0.5        1.00  2  2       8        0.00
#> 3                   0.8        4.00  2  0       8        0.25
#>   net_benefit_alert_all net_benefit_alert_none              rule estimate_kind
#> 1               0.21875                      0 alert if p >= 0.2      observed
#> 2              -0.25000                      0 alert if p >= 0.5      observed
#> 3              -2.12500                      0 alert if p >= 0.8      observed

The net-benefit display contains only the requested thresholds and does not interpolate a decision curve. Calibration groups are a display choice, with no sampling confidence intervals. In the workbench, changing these settings makes the previous evaluation stale until it is recalculated.

Missingness and uncertainty

The default missing-data policy is error. Choosing missing = "exclude" changes the evaluated sample and must be reported. Repeated observations from the same unit need an explicit cluster definition; treating them as independent can give misleading uncertainty estimates. Consult ?classification_uncertainty for supported interval and resampling settings.

evaluated$status
#> $code
#> [1] "ok"
#> 
#> $details
#> named list()
#> 
#> $reason
#> [1] ""
evaluated$limitations
#> $code
#> [1] "same_data_manual_cutoff"     "apparent_performance"       
#> [3] "fixed_cutoff_intervals"      "post_selection_not_computed"
#> 
#> $text
#> [1] "The manual cutoff is treated as chosen with access to these data (default); declare a predeclared rule or held-out tuning if that is true."           
#> [2] "The model is declared as developed on the evaluated data (evaluation split 'same_data'): all metrics, including the AUC, are apparent and optimistic."
#> [3] "Intervals of operating-point metrics are conditional on the fixed cutoff; for a data-driven cutoff they do not include the selection uncertainty."    
#> [4] "No post-selection (optimism-corrected or nested-resampling) interval is computed."
evaluated$uncertainty
#> $auc_method
#> [1] "delong"
#> 
#> $bootstrap_interval
#> [1] "percentile"
#> 
#> $bootstrap_metrics
#> NULL
#> 
#> $bootstrap_replicates
#> [1] 0
#> 
#> $bootstrap_rng
#> NULL
#> 
#> $bootstrap_strata
#> [1] "class"
#> 
#> $level
#> [1] 0.95
#> 
#> $proportion_methods
#> [1] "wilson"          "clopper_pearson"
#> 
#> $seed
#> NULL

A cutoff selected and evaluated on the same data can overstate performance. Prespecify a cutoff or separate tuning and evaluation samples. Small synthetic examples cannot establish transportability, clinical utility or a recommended operating threshold.

Keep the analysis record

path <- tempfile(fileext = ".rds")
saveRDS(spec, path)
restored <- readRDS(path)
stopifnot(identical(ig_hash(spec), ig_hash(restored)))

Use RDS for the classification specification: ig_from_json() currently imports power and report objects, not classification specifications. Retain the input definition, data provenance, result and software revision alongside the exported record. The getting-started tutorial explains the package’s typed outcomes and reproducibility workflow.

Portable reports use a separate privacy boundary. They omit individual rows, full score-derived ROC/PR curves and density tables. To include only the frozen calibration-bin means/counts and the requested net-benefit threshold aggregates, opt in explicitly:

envelope <- analysis_envelope(
  classification_evaluated_spec(probability_result), probability_result
)
report <- create_report_snapshot(envelope, include_probability_summaries = TRUE)
stopifnot(
  !report$privacy$raw_rows_included,
  identical(ig_hash(report), ig_hash(ig_from_json(ig_to_json(report))))
)

Small groups can still disclose sensitive information. Review aggregate content and user-supplied titles or model metadata before sharing. The report records the inclusion decision in schema 0.4; reports without these summaries retain their existing format unless another feature, such as a Boschloo lattice, requires the newer version.

sessionInfo()
#> R version 4.6.2 (2026-10-29)
#> Platform: x86_64-pc-linux-gnu
#> Running under: Ubuntu 26.04.1 LTS
#> 
#> Matrix products: default
#> BLAS:   /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3 
#> LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.32.so;  LAPACK version 3.12.0
#> 
#> locale:
#>  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
#>  [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
#>  [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
#>  [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
#>  [9] LC_ADDRESS=C               LC_TELEPHONE=C            
#> [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       
#> 
#> time zone: Etc/UTC
#> tzcode source: system (glibc)
#> 
#> attached base packages:
#> [1] stats     graphics  grDevices utils     datasets  methods   base     
#> 
#> other attached packages:
#> [1] inferguidR_0.0.0.9000 rmarkdown_2.32       
#> 
#> loaded via a namespace (and not attached):
#>  [1] digest_0.6.39      R6_2.6.1           fastmap_1.2.0      xfun_0.62         
#>  [5] maketools_1.3.2    cachem_1.1.0       knitr_1.52         htmltools_0.5.9   
#>  [9] buildtools_1.0.0   lifecycle_1.0.5    cli_3.6.6          jsonvalidate_1.5.0
#> [13] sass_0.4.10        jquerylib_0.1.4    compiler_4.6.2     sys_3.4.3         
#> [17] tools_4.6.2        curl_8.1.0         evaluate_1.0.5     bslib_0.12.0      
#> [21] Rcpp_1.1.2         yaml_2.3.12        otel_0.2.0         V8_8.2.1          
#> [25] jsonlite_2.0.0     rlang_1.3.0