---
title: "The cellspec 1.0 format"
output: rmarkdown::html_vignette
vignette: >
%\VignetteIndexEntry{The cellspec 1.0 format}
%\VignetteEngine{knitr::rmarkdown}
%\VignetteEncoding{UTF-8}
resource_files:
- figures/logo.svg
- figures/logo-dark.svg
---
```{r, include = FALSE}
knitr::opts_chunk$set(collapse = TRUE, comment = "#>")
library(cellspecR)
cols <- cs_vocabulary("columns")
voc <- cs_vocabulary("vocabularies")
checks <- cs_vocabulary("checks")
column_table <- function(component) {
tab <- cols[cols$component == component, c("column", "type", "requirement", "description")]
tab$column <- paste0("`", tab$column, "`")
rownames(tab) <- NULL
knitr::kable(tab, col.names = c("Column", "Type", "Requirement", "Meaning"))
}
values <- function(vocabulary) {
paste0("`", voc$value[voc$vocabulary == vocabulary], "`", collapse = ", ")
}
```
This document is the normative description of `cellspec`
`r cs_spec_version()`, the table format for segmented cells from multiplexed
tissue images. The key words **MUST**, **SHOULD** and **MAY** are used as in
RFC 2119. Every table below is generated from the objects that the validator
uses (`cs_vocabulary()`), so the text and the checks cannot drift apart.
This specification is shipped with cellspecR version
`r as.character(utils::packageVersion("cellspecR"))`.
## 1. Object
A `cellspec` object is a list of class `"cellspec"` with these components:
| Component | Type | Required | Content |
|---|---|---|---|
| `spec_version` | string | yes | `"`r cs_spec_version()`"` |
| `cells` | data frame | yes | one row per cell: identity and geometry (section 2) |
| `measurements` | double matrix | yes (may have 0 columns) | cells x features, rows in the order of `cells` (section 3) |
| `dictionary` | data frame | yes | one row per measurement column (section 4) |
| `images` | data frame | yes | one row per image (section 5) |
| `channels` | data frame | yes | one row per image and channel (section 5) |
| `provenance` | list | yes | origin of the data (section 6) |
| `adjacency` | data frame or `NULL` | no | cell-cell contacts (section 7) |
A whole slide carries up to about 1,000 numeric features for millions of
cells. A double matrix stores them compactly and maps directly to an assay of
a `SpatialExperiment`; `cells` stays a data frame because it holds mixed
types. Accessors return plain data frames. Rows of `measurements` are aligned
with `cells` by position; row names are not used.
```{r}
x <- cs_example()
x
```
## 2. Cells
```{r, echo = FALSE}
column_table("cells")
```
Rules:
- The key `(image_id, cell_id)` MUST be unique.
- `x` and `y` MUST be finite. `area`, when present, MUST be positive or `NA`.
- Identifiers are character columns, never row names.
- Additional columns are allowed and SHOULD be snake_case. The prefixes
`region__` and `boundary_distance__` are reserved for region annotations
and `cs__` for internal use. A layer name MUST match
`^[A-Za-z0-9][A-Za-z0-9_.-]*$`.
- `cells` MUST NOT contain clinical variables; `subject_id` belongs in
`images`.
## 3. Measurements
- A double matrix with one row per cell.
- Column names are feature identifiers and MUST equal
`dictionary$feature_id`, in the same order.
- Values are kept as exported by the source tool, in raw units. No
transformation is applied on import. `NaN` and infinite values from sources
are converted to `NA` and counted in `provenance$counts`.
## 4. Dictionary
```{r, echo = FALSE}
column_table("dictionary")
```
Controlled vocabularies:
- `kind`: `r values("kind")`.
- `compartment`: `r values("compartment")`, or `NA` for `other` features.
- `statistic` for intensity features: `r values("intensity_statistic")`.
- `statistic` for shape features: `r values("shape_statistic")`.
Unknown shape measurements are stored as `kind = "other"`, never dropped.
- `unit`: `r values("unit")`.
Feature identifiers are generated by readers and follow one rule per kind:
| Kind | Identifier | Example |
|---|---|---|
| intensity | `::` | `nucleus:FOXP3:mean` |
| shape | `:` | `cell:area` |
| other | `other:` | `other:detection_probability` |
Downstream code MUST select features through their dictionary columns, for
example with `cs_features()`, and not by splitting identifiers. Marker names
MUST be trimmed and MUST NOT contain `:`. Because identifiers follow the rule
above and are unique, no two features can describe the same kind, marker,
compartment and statistic.
```{r}
cs_features(x, kind = "intensity", marker = "FOXP3")
```
## 5. Images and channels
```{r, echo = FALSE}
column_table("images")
```
```{r, echo = FALSE}
column_table("channels")
```
Every intensity marker in the dictionary MUST appear in `channels$marker` for
every image of the object. A channel without measurements is allowed, for
example a nuclear stain used only for segmentation.
## 6. Provenance
`provenance` is a list with at least these fields:
| Field | Content |
|---|---|
| `producer` | `tool` and `version` of the software that measured the cells |
| `reader` | `package`, `version`, `adapter` and `adapter_version` that built the object |
| `created_utc` | creation time, `YYYY-MM-DDTHH:MM:SSZ` |
| `inputs` | one entry per input file: `path_basename`, `bytes`, `sha256` |
| `counts` | `rows_read`, `rows_kept`, `na_converted`, `nonfinite_converted` |
| `parameters` | reader parameters, e.g. where the pixel size came from |
| `history` | steps that changed content: `step`, `function`, `time_utc`, `details` |
Input paths are stored as file names plus hash. Full paths leak private
directory structures and are stored only when a user explicitly asks for it.
## 7. Adjacency (optional)
```{r, echo = FALSE}
column_table("adjacency")
```
`method` is one of `r values("adjacency_method")`. Each unordered pair of
cells appears at most once, ordered so that `cell_id_a` sorts before
`cell_id_b` in byte order, and both cells MUST exist in `cells`. Subsetting an
object drops contacts whose cells were removed.
## 8. Coordinate frame and units
- The origin is the top-left corner of the top-left full-resolution pixel;
x increases to the right and y increases downward (the QuPath convention).
- The centre of the pixel with 0-based index `i` lies at
`(i + 0.5) * pixel_size`.
- Centroids `x` and `y` are in micrometres. Readers that receive pixel
centroids MUST know the pixel size, from the file or from an argument, and
MUST stop otherwise.
- When a tool reports both micrometre and pixel centroids, readers check that
`abs(x - x_px * pixel_size) <= 0.5 * pixel_size` and stop on disagreement.
- The micro sign appears in exports as U+00B5, U+03BC, `um` or the mis-decoded
`µ`. Readers normalise these variants before matching headers and keep the
original header in `source_name`.
## 9. Directory layout
A `cellspec` directory holds one object on disk:
```
/
cells.parquet cells columns, then one column per feature_id
cellspec.json spec_version, column types, dictionary, images,
channels, provenance
adjacency.parquet optional
MANIFEST.sha256 " " for every file above
DONE written last; holds the SHA-256 of MANIFEST.sha256
```
- The cell table is stored as Parquet by default, which keeps every double
bit-exact. A tab-separated alternative (`cells.tsv.gz`,
`adjacency.tsv.gz`) is readable by any tool; its numbers carry 15
significant digits, so values can differ from the originals in the last
representable digits.
- `cellspec.json` validates against the JSON Schema shipped in
`system.file("schema", "cellspec-1.0.0.schema.json", package = "cellspecR")`.
- Writers stage the directory under a temporary name and rename it at the
end, so a directory without `DONE` was not completely written.
- Readers refuse a directory without `DONE`, with a manifest mismatch, or with
an unsupported major `spec_version`.
## 10. Validation rules
`cs_validate()` checks an object against the rules above. Structure checks
implement the MUST rules and fail; semantic checks flag implausible content
and warn.
```{r, echo = FALSE}
tab <- checks
tab$check <- paste0("`", tab$check, "`")
knitr::kable(tab, col.names = c("Check", "Level", "Rule"))
```
## 11. Versioning
- `spec_version` follows semantic versioning. Minor versions only add
optional columns or components.
- A reader of major version 1 accepts every 1.y object whose minor version is
not newer than its own.
- Readers of tool exports carry their own `adapter_version`, which changes
whenever their output for the same input changes.