---
title: "Validation and reliability"
output: rmarkdown::html_vignette
vignette: >
%\VignetteIndexEntry{Validation and reliability}
%\VignetteEngine{knitr::rmarkdown}
%\VignetteEncoding{UTF-8}
resource_files:
- figures/logo.svg
- figures/logo-dark.svg
---
```{r setup, include=FALSE}
library(hypermodR)
knitr::opts_chunk$set(collapse = TRUE, comment = "#>", error = FALSE)
```
This exact-value example evaluates four synthetic classification observations.
Subjects are the independent units; the score is a pooled descriptive estimand,
not a clinical or external validation claim.
```{r known-metrics}
task <- hm_task("truth", "classification", "spectrum", "subject_id")
reference <- data.frame(
observation_id = letters[1:4],
subject_id = paste0("unit_", 1:4), truth = c("a", "a", "b", "b")
)
pred <- hm_prediction(c("a", "b", "b", "b"), task, letters[1:4],
probabilities = matrix(c(.8, .2, .4, .6, .3, .7, .1, .9), 4, 2,
byrow = TRUE,
dimnames = list(NULL, c("a", "b"))
), class_map = c("a", "b")
)
e <- hm_evaluate(pred, reference)
if (!isTRUE(all(
unname(e$metrics["accuracy"]) == .75,
isTRUE(all.equal(unname(e$metrics["brier"]), .25))
))) {
cli::cli_abort("Example assertion failed.", class = "hm_example_error")
}
e
hm_reliability(pred, reference)
```
`hm_split()` connects biological/source identities before assignment. Frame
windows from the same source cannot become independent units. Nested tuning,
preprocessing, early stopping and calibration belong inside outer training.
Temporal splits bind the selected time column and its observation identities.
Regenerate a split after changing timestamps, cutoff, look-back or embargo;
fitting rejects a stale clock or altered temporal partition.
`hm_calibrate()` requires an explicit inner fit and outer split;
`hm_apply_calibration()` applies its frozen temperature/abstention settings.
`hm_active_learning()` proposes training-pool annotation review, not test-label
access or automatic truth changes.
`hm_bootstrap()` retains whole-unit draws and reports intervals conditional on
fixed predictions. Training and model selection uncertainty are excluded.
Missing-class/nonfinite draws invalidate intervals; retain the draw accounting.
The equal-class/subject/acquisition accuracy estimand differs from pooled
accuracy. Probability calibration is not evidence of OOD detection.
`hm_benchmark_protocol()` validates protocol admission for the central
[Spectralverse](https://spectralverse-heidelberg.org) reference programme:
[HeiPorSPECTRAL](https://spectralverse-heidelberg.org/HeiPorSPECTRAL/) primary
anatomy, [xeno-spectral](https://spectralverse-heidelberg.org/xeno-spectral/)
development and separately admitted [ManUSPECTRAL](https://spectralverse-heidelberg.org/manuspectral/)
human targets.
Known/prior target access remains development or unresolved. External datasets,
source rights, transfer performance and clinical intended use need independent
evidence. No protocol validation alone establishes scientific qualification.
## Reconstruction coordinates
Preprocessing can change both units and the number of spectral features. A
reconstruction model predicts in its fitted recipe coordinates. With a nonempty
recipe, `hm_evaluate()` requires an explicitly transformed target and matching
space declaration. Copying the declaration alone does not transform a target.
The resulting error and angle describe these coordinates, not native spectra.
Named matrix outputs and targets must also have identical column names/order.
```{r reconstruction-coordinates}
data <- hm_example()
task <- hm_task(NULL, "reconstruction", "spectrum", "subject_id")
fold <- hm_split(data, task, folds = 3L, inner_folds = 2L)$folds[[1L]]
workflow <- hm_workflow(task, hm_recipe(list(list(type = "scale"))),
hm_model_spec("pca", task, list(rank = 8L)))
fit <- hm_fit(workflow, data, fold)
heldout_ids <- data@observations$observation_id[fold$test]
heldout <- hm_spectra(data, list(observation_id = heldout_ids))
prediction <- hm_predict(fit, heldout)
reference <- heldout$observations
attr(reference, "target") <- hm_bake(fit$recipe, heldout$values)
attr(reference, "target_space") <- prediction$provenance$output_space
reconstruction <- hm_evaluate(prediction, reference)
reconstruction
reconstruction$interpretation
```
Prediction also requires the same declared radiometric input unit as fitting.
Calibration identifiers may differ across acquisitions; this check does not
qualify a device or prove that its declared units/calibration are correct.
Previously saved fits without an input-unit contract must be refitted.
Survival risk predictions are finite numeric vectors, with complete numeric
nonnegative event times and binary status in the evaluation reference.
