Images and backends

Image handles and windowed readers

Reading an image returns an annot_image with dimensions, pyramid levels, bands, pixel size and a tile accessor, at_tile(). The ENVI, TIVITA SpecCube and OME readers open only the metadata and read the requested windows from the file. The plain TIFF and raster readers read the complete image when it is opened; TIFF decoding runs in a separate R process, which isolates native decoder failures but does not reduce the memory the decoded image needs. at_image_capabilities() reports the effective strategy, including images that still hold a pixel snapshot in memory. Allow additional memory for decoder buffers, returned tiles and the tile cache.

img <- at_example_image("tissue")
img
#> <annot_image> example_tissue.png
#> backend: "raster"  |  dtype: "uint8"
#> size: 512 x 512 px  |  levels: 1  |  bands: 3
#> pixel size: 1 x 1 px
dim(img)
#> [1] 512 512   3

The dim() of a handle is c(width, height, bands). Printing it summarises the backend, data type and extent; the metadata accessors use the fields already read.

str(at_image_capabilities(img))
#> List of 5
#>  $ metadata_only: logi FALSE
#>  $ window_read  : logi FALSE
#>  $ samples      : chr "display-converted-0-255"
#>  $ io           : chr "memory"
#>  $ lifecycle    : chr "retained pixel snapshot"
at_image_capabilities(at_example_image("cube"))$window_read
#> [1] TRUE

A handle also exposes accessors for everything an annotation workflow needs: at_n_bands() and at_bands() for the channels, at_pixel_size() for physical scale, and at_meta() for backend-specific keys such as the source path and data type. Each simply reads a field of the handle — no I/O is involved.

Pyramid levels

Slide and multiplex formats store the same scene at several resolutions — a pyramid. Level 0 is full resolution; each higher level is a coarser overview. The multiplex example is a three-level pyramidal TIFF.

mx <- at_example_image("multiplex")
at_n_levels(mx)
#> [1] 3
at_is_pyramidal(mx)
#> [1] TRUE

levels <- 0:(at_n_levels(mx) - 1L)
data.frame(
  level  = levels,
  width  = vapply(levels, function(l) at_dims(mx, l)[1], integer(1)),
  height = vapply(levels, function(l) at_dims(mx, l)[2], integer(1))
)
#>   level width height
#> 1     0   512    512
#> 2     1   256    256
#> 3     2   128    128

Levels matter for two reasons. First, an overview lets you display and navigate the whole image at reduced resolution; windowed readers can do this without reading full-resolution pixels. Second, an ROI drawn on a coarse level records the level it was created at, so its coordinates can be rescaled to level 0 (see at_roi_rescale() and at_transform()). Every geometry in hyperannotatR is anchored to a pyramid level; the coordinate convention is image pixels with the origin at the top-left and y increasing downward.

Backends

A backend is the reader for a particular family of formats. The registry maps format names to four functions — read metadata, fetch a tile, detect a file, and report whether its optional dependencies are installed — so new formats can be added without touching core code.

at_backend_list()
#> # A tibble: 6 × 6
#>   name    description             extensions available metadata_only window_read
#>   <chr>   <chr>                   <chr>      <lgl>     <lgl>         <lgl>      
#> 1 cuvis   Cubert hyperspectral (… cu3, cu3s  FALSE     FALSE         FALSE      
#> 2 envi    ENVI header + binary c… hdr, dat,… TRUE      TRUE          TRUE       
#> 3 ometiff OME-TIFF and qptiff (R… ome.tif, … TRUE      FALSE         FALSE      
#> 4 raster  PNG, JPEG, and single-… png, jpg,… TRUE      FALSE         FALSE      
#> 5 tiff    Pyramidal / multi-dire… tif, tiff  TRUE      FALSE         FALSE      
#> 6 tivita  Diaspective Vision Tiv… dat, hdr   TRUE      TRUE          TRUE

The available column is evaluated lazily: a backend for a format whose support package is absent still appears in the list but reads nothing until you install the package. This keeps the package usable with a minimal set of dependencies, and means a missing optional package never breaks loading — only reading that particular format. When you call at_read_image() without naming a backend, the registry tries the available backends in priority order and picks the first that recognises the file; more specific readers are tried before general ones, so an OME-TIFF is not claimed by the plain raster reader. You can always bypass detection by passing backend = explicitly.

Plain TIFF remains the explicit eager fallback. Its pyramid directories are not silently replaced by a first-directory GDAL view. The qualified OME path uses BioFormats subsets with normalization disabled, preserving image orientation and actual per-level dimensions. Availability is not qualification of every vendor/qptiff layout or deployment. No vendor whole-slide fixture is qualified.

Raw samples and input validation

ENVI reads validate positive integer dimensions, interleave, byte order, sample type, header offset and payload length before allocation. Header offsets skip leading bytes; trailing metadata is permitted. Signed int32 image samples, including -2147483648, remain exact doubles. NPY masks accept 2-D scalar integer arrays in C or Fortran order, with version 1, 2 or 3 headers. Zero-sized masks, structured/object dtypes, malformed headers, truncated payloads and samples outside [-2147483647, 2147483647] are rejected. R reserves -2147483648 for integer NA, so that value cannot be an integer mask code. Signed and unsigned 64-bit inputs are accepted only when every complete value fits this mask range. A NPY reader consumes the declared array and permits subsequent bytes/arrays.

Binary file sizes/offsets must fit exact double integer arithmetic (at most 2^53-1 bytes), and each declared dimension and each returned window’s element count must fit R’s integer indexing limit. ENVI headers have a fixed 1 MiB limit; NPY headers have a separate 1 MiB default configurable with hyperannotatR.max_npy_header_bytes. Array budgets and their limits are described under Memory discipline below.

The tiff backend preserves raw grayscale uint8/uint16, signed int16 and float32 samples, and RGB uint16 samples. Sample type and photometric metadata determine decoding; integer display normalisation is reversed without scaling float samples. Unsupported encodings, including integer32/float64 TIFF in the qualified minimal tiff installation, fail explicitly. Independent equal-size TIFF directories are ambiguous and are rejected instead of discarding planes. A consistent decreasing pyramid retains its actual dimensions at every level.

OME-TIFF and qptiff detection requires RBioFormats and xml2. The OME reader normalises [x,y,channel] to [y,x,band], preserves actual resolution dimensions, and reads channel names, emission wavelengths and physical pixel sizes from OME-XML. Missing/incomplete calibration is labelled missing in image metadata; its pixel grid remains in px. Missing wavelengths remain NULL. Multiple series require at_read_image(path, series = ...); non-singleton Z/time axes are rejected and must be selected/exported as a single plane beforehand.

Automatic TIFF/OME detection errors when the required raw reader is absent. Explicit backend = "raster" selects the separate display-conversion semantics, which can lose sample depth and format metadata. TIFF mask imports likewise require the raw TIFF reader and a single finite integer plane.

Reopening saved reader contracts

ENVI source descriptors record the canonical paths, byte sizes and modification times of both the header and payload, regardless of which path was opened. TIVITA SpecCubes also record the selected .tivita.json profile when a sidecar was used. Source checks verify both the recorded files and current discovery: a new higher-priority header, payload or profile, or a retargeted companion symlink, cannot silently select different decoding inputs. A newly introduced sidecar invalidates a cube previously opened with the default profile. Explicit at_tivita_profile() arguments continue to ignore incidental sidecars.

at_inspect_source() reports changed, missing or unresolved inputs without reading pixels. File-backed tile requests check the source before consulting the cache and again before caching a newly read result. These size/mtime checks detect ordinary replacements, not edits that preserve both values. Concurrent file writers are unsupported.

Saved TIFF/OME images record a stable semantic contract (axes = "yxb", samples = "raw-scalar-v1") in image metadata and durable source descriptors. This describes pixel meaning, independently of package release or eager/windowed execution. Mask sidecars retain it; masks with missing affected contracts or conflicting contracts need an explicit alignment assertion rather than being certified as source-aligned. Custom backends without this marker retain their existing descriptor comparison semantics.

On reopen, dimensions, levels, band count, dtype, channel names, wavelengths, units, pixel calibration and the reader contract must agree. TIVITA also checks the effective profile digest and value unit. A stale or ambiguous lightweight project still loads its annotations and IDs, but pixel access reports the mismatch. Historical ENVI descriptors without both companion files, and TIVITA sidecar descriptors without the recorded profile dependency, remain unverified. Today’s files are never used to backfill historical provenance automatically.

After checking that the cube, profile and reader options are the intended ones, inspect and explicitly revalidate the source:

at_inspect_source(project)
project <- at_relink_source(project, project$image$source)

Relinking refreshes reader metadata and retains its geometry/wavelength compatibility guard; an old transposed OME geometry or changed wavelength declaration is not guessed or automatically registered. Caller-modified calibration also requires explicit review if it differs from the reopened reader. Full legacy snapshots whose in-memory pixel handles remain available retain those original pixels, including when saved again. Releasing such a snapshot may make its pixels unavailable if the source cannot be revalidated.

The universal tile accessor

Whatever the backend, pixels come out through one function, at_tile(), and always as a three-dimensional [y, x, band] numeric array — even for a single band. You choose the pyramid level, an optional pixel window, and an optional subset of bands.

tile <- at_tile(mx, level = 0, xrange = c(1, 64), yrange = c(1, 64))
dim(tile)
#> [1] 64 64  4

The returned array contains only that 64 x 64 window of every band. Eager backends have already read the underlying image; the window-capable backends read only the requested runs or subsets. Band selections are finite whole indices, keep their order and may repeat. To request a thumbnail, read the coarsest level:

overview <- at_tile(mx, level = 2)
dim(overview)
#> [1] 128 128   4

dim(at_tile(mx, level = 2, bands = 1))
#> [1] 128 128   1

Spectral cubes are handled by the very same accessor; their band metadata simply carries wavelengths, which you can read back with the image accessors.

cu <- at_example_image("cube")
at_is_spectral(cu)
#> [1] TRUE
head(at_bands(cu), 3)
#> # A tibble: 3 × 7
#>   index name   wavelength unit   fwhm order wavelength_status
#>   <int> <chr>       <dbl> <chr> <dbl> <int> <chr>            
#> 1     1 Band 1       450  nm       NA     1 ok               
#> 2     2 Band 2       462. nm       NA     2 ok               
#> 3     3 Band 3       473. nm       NA     3 ok
range(at_wavelengths(cu))
#> [1] 450 900

at_band_view() also requires finite numeric whole band indices: 1.9, "1" and factors are rejected before integer conversion. Its wavelength_window_mean operation requires finite numeric bounds with wavelength_min <= wavelength_max. Band views are display products; they do not replace the stored samples used for extraction.

Raw-to-reflectance conversion requires explicitly supplied finite numeric white and dark reference vectors. Each reference is a scalar or has one value per band; per-band values follow the third dimension of a [y,x,band] array, the columns of a pixel-by-band matrix, or the elements of a spectrum. Factors, strings, complex values, empty vectors and nonfinite references fail with CALIBRATION_INVALID, rather than being coerced to different numbers. A zero white-minus-dark denominator yields NA.

at_convert_values(c(100, 550, 1000), from = "raw", to = "reflectance",
                  calibration = list(white = 1000, dark = 100))
#> [1] 0.0 0.5 1.0
#> attr(,"value_unit")
#> [1] "reflectance"
#> attr(,"conversion")
#> [1] "raw->reflectance"

Registering a custom backend

Adding support for a new format means supplying the four functions and calling at_backend_register(). The read_fn returns an annot_image — a classed list of metadata plus a handle for the accessor — and the tile_fn turns a request into a [y, x, band] array. Here is a complete, self-contained backend for a toy “constant image” format that stores only a size and a fill value, and synthesises pixels on demand.

const_read <- function(path, ...) {
  spec <- readRDS(path)                 # list(width, height, bands, value)
  structure(
    list(
      source     = path,
      backend    = "const",
      dims       = c(spec$width, spec$height),
      n_levels   = 1L,
      level_dims = list(c(spec$width, spec$height)),
      n_bands    = spec$bands,
      band_names = paste0("Band ", seq_len(spec$bands)),
      pixel_size = c(1, 1),
      pixel_unit = "px",
      dtype      = "double",
      handle     = list(value = spec$value),   # no pixels, just the fill value
      meta       = list()
    ),
    class = "annot_image"
  )
}

const_tile <- function(img, level, xrange, yrange, bands) {
  nx <- xrange[2] - xrange[1] + 1L
  ny <- yrange[2] - yrange[1] + 1L
  nb <- if (is.null(bands)) img$n_bands else length(bands)
  array(img$handle$value, dim = c(ny, nx, nb))
}

at_backend_register(
  name         = "const",
  read_fn      = const_read,
  tile_fn      = const_tile,
  detect_fn    = function(path) grepl("\\.const$", path, ignore.case = TRUE),
  available_fn = function() TRUE,
  description  = "Constant-value demo backend",
  extensions   = "const"
)

The new backend now behaves like any other — it appears in the registry, is auto-detected from the file, and serves tiles through at_tile().

path <- tempfile(fileext = ".const")
saveRDS(list(width = 64L, height = 48L, bands = 2L, value = 0.5), path)

cimg <- at_read_image(path)
cimg
#> <annot_image> file136616262ab9.const
#> backend: "const"  |  dtype: "double"
#> size: 64 x 48 px  |  levels: 1  |  bands: 2
#> pixel size: 1 x 1 px
dim(at_tile(cimg, xrange = c(1, 8), yrange = c(1, 8)))
#> [1] 8 8 2

Memory discipline

The limits below check particular arrays. They do not bound total process memory, temporary copies, native decoder buffers or filesystem cache use.

Set a positive finite hyperannotatR.max_read_bytes value when a trusted array needs a larger limit. A valid windowed source is not rejected solely because its complete cube exceeds that budget. Plain TIFF and raster image opening is eager and has no predecode check against these tile/read budgets: the complete image or pyramid must fit in memory before a tile is requested. The isolated TIFF decoder protects the calling process from decoder failures; it does not make TIFF opening a windowed or memory-budgeted operation.

Binary readers restrict spatial reads to the requested window, but encoded input buffers can include unrequested bands. ENVI BSQ uses contiguous runs per selected band. BIL uses band runs or full-band row chunks; BIP reads every band for the requested rows and columns. SpecCube reads every band for the requested rows of each selected column. Conversion buffers and reordered output copies need additional memory beyond the selected-band array budget. OME delegates subset decoding to BioFormats; compressed blocks may exceed the requested window, and the R array budget does not control its native buffers.

Full-image at_tile() still requests a full image. Extraction reads one ROI bounding-box tile, without within-ROI streaming. at_mask() checks a full-grid estimate of 32 bytes per pixel against hyperannotatR.max_read_bytes before rasterisation. This covers an array-work allowance, not native/geometry memory; at_mask_stack() can retain several layer masks and a combined array. Preview, queue and export memory require their own limits; windowing alone does not make every annotation workflow bounded.

at_close_image(img) returns a copy with its handle released and a fresh runtime identity; assign it back. The first subsequent tile reopens a verified durable descriptor. Repeated identical requests on that closed image can then use its cache without reopening again, after validating source freshness. Closing does not purge the shared cache, whose byte limit still governs retained entries; the fresh identity prevents reuse of the previous image’s tiles and counters. Built-in window handles contain only paths/layout metadata, open each binary connection per read and close it on exit, including errors. No native pointer is serialized. Registered backends can supply optional capabilities and close_fn=function(img) arguments; existing positional registration calls remain valid. Per-image meta$capabilities may refine registry declarations. A retained in-memory array always reports a memory snapshot through at_image_capabilities().

at_read_stats() counts backend tile reads, payload bytes and decoded-array bytes under the caller’s image identity, including after a closed image reopens. Cache hits do not count as backend reads; header/metadata I/O is not included in the payload counter. This example makes two identical requests but records one backend tile read:

cu <- at_close_image(at_example_image("cube"))
invisible(at_read_stats(cu, reset = TRUE))
invisible(at_tile(cu, xrange = c(1, 2), yrange = c(1, 2), bands = 1))
invisible(at_tile(cu, xrange = c(1, 2), yrange = c(1, 2), bands = 1))
at_read_stats(cu)[c("tiles", "file_bytes", "decoded_bytes")]
#> $tiles
#> [1] 1
#> 
#> $file_bytes
#> [1] 8
#> 
#> $decoded_bytes
#> [1] 32

For qualification, internal .read_telemetry_reset() / .read_telemetry() counters can bracket open and tile operations. binary_bytes counts bytes actually returned by instrumented binary/header reads and binary_calls counts those calls. Native metadata/window calls and returned decoded-double bytes are separate; native encoded file traffic is unknown (NA), never estimated from the decoded request. These counters are not OS cache misses, disk throughput or RSS. Measure process memory separately from before open, including native startup.

Next, see the The annotation model vignette to learn how ROIs, layers and projects build on these image handles.

Quantitative extraction contracts

at_extract(project, stat = "all", nonfinite = "omit") computes each ROI and band separately using finite samples only. n_px (and statistic n) counts selected support; n_valid counts finite samples and n_invalid counts NA, NaN and infinities. The default nonfinite = "propagate" returns NA summary statistics for any affected ROI/band. "error" rejects selected nonfinite samples. Empty support emits no rows, a singleton has NA sample SD, and an all-invalid selection under omit has NA statistics including sum. Overlapping ROIs count shared pixels separately; ROI rows must not be pooled as independent pixels or images. The output analysis attribute records source, level, bands, policy and limitations.

Wavelength values and their unit come from at_bands(). They are not silently converted or relabelled: missing ENVI units remain unknown. Spectrum output orders by ROI, unit, coordinate and band. at_plot_spectrum() labels the declared spectral coordinate unit and rejects mixed units until explicitly converted. Display contrast does not change extracted tiles. The raster fallback already converts source samples to a 0–255 display convention, so its extraction is not an assertion of original high-bit-depth intensity preservation.

at_roi_area(roi, units = "physical", image = image) uses the image’s level-zero pixel calibration and actual pyramid dimensions; physical area is invariant to reporting level. The result is expressed in the square of image$pixel_unit. Unknown or pixel-only calibration rejects. Level-zero legacy ROI pixel_size attributes remain supported in squared calibration units; nonzero stored ROI levels require explicit pixel_size_level and pixel_unit instead of an inferred calibration. Geometric area differs from the number of raster-selected pixels.

at_mask_agreement() validates annotated masks’ grids, source descriptors, encodings and codebooks. alignment = "assert" records an explicit registration assertion; code_alignment = "by-label" separately aligns categorical or bitfield codes by semantic labels. Instance alignment uses stable ROI IDs via "by-id", never repeated class names. Display labels cannot authorize remapping. Binary and categorical metrics compare equality; bitfield metrics compare memberships, with exact-set accuracy and no categorical kappa. Declared absent classes have NA Dice/IoU and do not contribute to macro means. A background-only pair has accuracy 1 and undefined (NA) kappa. Plain matrices remain explicit-code legacy categorical inputs and reject fractional, nonfinite or unrepresentable values.