---
title: "Building a Corpus from Files and APIs"
output: rmarkdown::html_vignette
vignette: >
%\VignetteIndexEntry{Building a Corpus from Files and APIs}
%\VignetteEngine{knitr::rmarkdown}
%\VignetteEncoding{UTF-8}
resource_files:
- figures/logo.svg
- figures/logo-dark.svg
---
[](https://github.com/CTTIR/scimapR/actions/workflows/R-CMD-check.yaml)
[](https://cttir.github.io/scimapR/)
[](https://CRAN.R-project.org/package=scimapR)
[](https://app.codecov.io/gh/CTTIR/scimapR?branch=main)
[](https://cran.r-project.org/package=scimapR)
[](https://cran.r-project.org/package=scimapR)
[](https://opensource.org/licenses/MIT)
[](https://lifecycle.r-lib.org/articles/stages.html#experimental)
```{r, include = FALSE}
knitr::opts_chunk$set(
collapse = TRUE,
comment = "#>"
)
```
```{r setup, include = FALSE}
library(scimapR)
```
## Bring your own relational data (recommended)
If your data is already tabular -- a manual tracker, an institutional export, a
spreadsheet of works -- the most robust entry point is
`sm_corpus_from_tables()`. It validates the required columns, coerces types,
fills missing optional tables with correctly-typed 0-row tibbles, and returns a
ready `sm_corpus`, side-stepping format-specific parsers entirely. A works-only
corpus is perfectly valid:
```{r from-tables}
works <- data.frame(
work_id = paste0("W", 1:3),
title = c("Spatial transcriptomics in cancer",
"Immune checkpoint resistance",
"A biomarker discovery cohort"),
year = c("2019", "2020", "2021"), # character -> coerced to integer
doi = paste0("10.1234/example.", 1:3),
cited_by_count = c(12, 5, 1)
)
corpus <- sm_corpus_from_tables(list(works = works))
corpus
```
You can pass any of `works`, `authorships`, `sources`, `institutions`,
`references`, `concepts`, etc. as named list elements.
### Materialising cached enrichment
Once you have a corpus, fold cached enrichment (a metrics table, a citation
cache, ...) into it with `sm_materialise()` rather than hand-writing the join.
It matches by key, fills `NA` cells (or replaces with `overwrite = TRUE`), and
returns a schema-valid corpus --- avoiding the classic `bind_rows()` bug where a
`NULL` element silently becomes a logical column.
```{r materialise}
metrics <- data.frame(work_id = corpus$works$work_id,
cnci = c(1.4, 0.8, 1.1))
corpus <- sm_materialise(corpus, sources = list(works = metrics))
"cnci" %in% names(corpus$works)
```
## File ingestion
scimapR reads 12 bibliographic formats with native clean-room parsers.
```{r files}
# List available example files
sm_example_files()
```
```{r read-bib}
# Read a BibTeX file
bib_path <- sm_example_files("example.bib")
corpus_bib <- sm_read_bib(bib_path)
print(corpus_bib)
```
```{r read-ris}
# Read an RIS file
ris_path <- sm_example_files("example.ris")
corpus_ris <- sm_read_ris(ris_path)
nrow(corpus_ris$works)
```
## Combining multiple sources
```{r combine}
combined <- sm_build_corpus(corpus_bib, corpus_ris, dedupe = TRUE)
nrow(combined$works)
```
## Auto-detection
```{r auto}
corpus_auto <- sm_read_auto(bib_path)
nrow(corpus_auto$works)
```
## API fetching
For API-based ingestion, set your email for polite pool access:
```r
Sys.setenv(SCIMAPR_MAILTO = "your.email@example.com")
# Fetch from OpenAlex
corpus <- sm_fetch_openalex(
query = "spatial transcriptomics colorectal cancer",
n_max = 100
)
```
All API fetchers populate the provenance table for reproducibility tracking.
## Deduplication
```{r dedupe}
corpus <- sm_example_corpus()
deduped <- sm_dedupe(corpus)
```
## Validation
```{r validate}
issues <- sm_validate(corpus)
nrow(issues)
```
