Validation and reliability

This exact-value example evaluates four synthetic classification observations. Subjects are the independent units; the score is a pooled descriptive estimand, not a clinical or external validation claim.

task <- hm_task("truth", "classification", "spectrum", "subject_id")
reference <- data.frame(
  observation_id = letters[1:4],
  subject_id = paste0("unit_", 1:4), truth = c("a", "a", "b", "b")
)
pred <- hm_prediction(c("a", "b", "b", "b"), task, letters[1:4],
  probabilities = matrix(c(.8, .2, .4, .6, .3, .7, .1, .9), 4, 2,
    byrow = TRUE,
    dimnames = list(NULL, c("a", "b"))
  ), class_map = c("a", "b")
)
e <- hm_evaluate(pred, reference)
if (!isTRUE(all(
  unname(e$metrics["accuracy"]) == .75,
  isTRUE(all.equal(unname(e$metrics["brier"]), .25))
))) {
  cli::cli_abort("Example assertion failed.", class = "hm_example_error")
}
e
#>          accuracy balanced_accuracy          mean_iou             brier 
#>         0.7500000         0.7500000         0.5833333         0.2500000 
#>          log_loss 
#>         0.4003674
hm_reliability(pred, reference)
#> $bins
#>          bin n confidence accuracy
#> 1    [0,0.1] 0         NA       NA
#> 2  (0.1,0.2] 0         NA       NA
#> 3  (0.2,0.3] 0         NA       NA
#> 4  (0.3,0.4] 0         NA       NA
#> 5  (0.4,0.5] 0         NA       NA
#> 6  (0.5,0.6] 1        0.6        0
#> 7  (0.6,0.7] 1        0.7        1
#> 8  (0.7,0.8] 1        0.8        1
#> 9  (0.8,0.9] 1        0.9        1
#> 10   (0.9,1] 0         NA       NA
#> 
#> $coverage_risk
#>   threshold coverage risk
#> 1       0.0     1.00 0.25
#> 2       0.6     1.00 0.25
#> 3       0.7     0.75 0.00
#> 4       0.8     0.50 0.00
#> 5       0.9     0.25 0.00
#> 6       1.0     0.00   NA
#> 
#> $scores
#>          accuracy balanced_accuracy          mean_iou             brier 
#>         0.7500000         0.7500000         0.5833333         0.2500000 
#>          log_loss 
#>         0.4003674 
#> 
#> $ood
#> [1] "Confidence is not a validated OOD detector; no OOD/safety claim."

hm_split() connects biological/source identities before assignment. Frame windows from the same source cannot become independent units. Nested tuning, preprocessing, early stopping and calibration belong inside outer training. Temporal splits bind the selected time column and its observation identities. Regenerate a split after changing timestamps, cutoff, look-back or embargo; fitting rejects a stale clock or altered temporal partition. hm_calibrate() requires an explicit inner fit and outer split; hm_apply_calibration() applies its frozen temperature/abstention settings. hm_active_learning() proposes training-pool annotation review, not test-label access or automatic truth changes.

hm_bootstrap() retains whole-unit draws and reports intervals conditional on fixed predictions. Training and model selection uncertainty are excluded. Missing-class/nonfinite draws invalidate intervals; retain the draw accounting. The equal-class/subject/acquisition accuracy estimand differs from pooled accuracy. Probability calibration is not evidence of OOD detection.

hm_benchmark_protocol() validates protocol admission for the central Spectralverse reference programme: HeiPorSPECTRAL primary anatomy, xeno-spectral development and separately admitted ManUSPECTRAL human targets. Known/prior target access remains development or unresolved. External datasets, source rights, transfer performance and clinical intended use need independent evidence. No protocol validation alone establishes scientific qualification.

Reconstruction coordinates

Preprocessing can change both units and the number of spectral features. A reconstruction model predicts in its fitted recipe coordinates. With a nonempty recipe, hm_evaluate() requires an explicitly transformed target and matching space declaration. Copying the declaration alone does not transform a target. The resulting error and angle describe these coordinates, not native spectra. Named matrix outputs and targets must also have identical column names/order.

data <- hm_example()
task <- hm_task(NULL, "reconstruction", "spectrum", "subject_id")
fold <- hm_split(data, task, folds = 3L, inner_folds = 2L)$folds[[1L]]
workflow <- hm_workflow(task, hm_recipe(list(list(type = "scale"))),
                        hm_model_spec("pca", task, list(rank = 8L)))
fit <- hm_fit(workflow, data, fold)
heldout_ids <- data@observations$observation_id[fold$test]
heldout <- hm_spectra(data, list(observation_id = heldout_ids))
prediction <- hm_predict(fit, heldout)
reference <- heldout$observations
attr(reference, "target") <- hm_bake(fit$recipe, heldout$values)
attr(reference, "target_space") <- prediction$provenance$output_space
reconstruction <- hm_evaluate(prediction, reference)
reconstruction
#>              rmse               mae              bias    spectral_angle 
#>      5.004711e-16      3.358960e-16     -1.139429e-17      2.441295e-09 
#> zero_norm_spectra 
#>      0.000000e+00
reconstruction$interpretation
#> [1] "Error and angle in fitted recipe coordinates, not native spectral error; zero-norm outputs remain unavailable."

Prediction also requires the same declared radiometric input unit as fitting. Calibration identifiers may differ across acquisitions; this check does not qualify a device or prove that its declared units/calibration are correct. Previously saved fits without an input-unit contract must be refitted. Survival risk predictions are finite numeric vectors, with complete numeric nonnegative event times and binary status in the evaluation reference.

- Reconstruction coordinates