Package 'scimapR'

Description

Generic function to convert various data types to an sm_corpus object.

Usage

as_sm_corpus(x, ...)

## S3 method for class 'sm_corpus'
as_sm_corpus(x, ...)

## S3 method for class 'data.frame'
as_sm_corpus(x, source_label = "data.frame", ...)

## S3 method for class 'list'
as_sm_corpus(x, ...)

## S3 method for class 'bibliometrixDB'
as_sm_corpus(x, source_label = "bibliometrix-import", ...)

Arguments

Value

An sm_corpus object.

bibliometrix data frames

as_sm_corpus() reads a data frame made by bibliometrix::convert2df() or sm_to_bibliometrix() (class bibliometrixDB), and the file readers with engine = "bibliometrix" use the same conversion:

• works from TI, AB (TI_raw and AB_raw where present, which keep the case of the file), PY, DT, TC, LA, DI and PM;

• authors and authorships from AU and AF, with one authorship row per author and address: the names in the brackets of C1 (C1raw where present) link authors to addresses. An address without names, as in older records, belongs to every author of the work; one whose names designate none of the authors belongs to the author of a single-author work and else to nobody. Authors are told apart by the full names of AF; an entry whose given names are only initials, or that has no AF, belongs to the author with that surname and those initials when there is just one such author with a full name. RP gives is_corresponding: the authors it names before "(corresponding author)" or "(reprint author)", in any case, are TRUE on all their rows and the other authors of the work FALSE. Namesakes (the same surname and initials) are told apart by the address that RP gives with the name, and stay NA when it does not tell them apart; a work without RP, or whose RP designates none of its authors for certain, gets NA (unknown);

• sources from SO, SN, EI, J9 and JI (issn_l, issn, abbreviation, iso_abbreviation);

• references from CR, with the DOI of each item that has one;

• concepts from DE (DE_raw where present), ID, SC and WC, in the vocabularies author-keywords, the indexer vocabulary of the source, wos-subject-category and wos-category. ID is read as keywords-plus for Web of Science data (DB = "ISI"), mesh for PubMed, Dimensions, Lens and Cochrane, author-keywords for BibTeX and RIS files (where bibliometrix stores the keywords of the file in ID) and index-keywords otherwise;

• provenance with UT as source_id_external.

See Also

Other corpus: is_sm_corpus(), sm_bind_corpora(), sm_build_corpus(), sm_concept_vocabularies(), sm_corpus(), sm_corpus_from_tables(), sm_dedupe(), sm_link_references(), sm_materialise(), sm_oa_statuses(), sm_save_corpus(), sm_validate(), validate_sm_corpus()

Examples

corpus <- sm_example_corpus()
as_sm_corpus(corpus)
as_sm_corpus(corpus$works)

Description

Automatically generate a suitable plot for an sm_corpus based on the requested type.

Usage

## S3 method for class 'sm_corpus'
autoplot(
  object,
  type = c("production", "landscape", "thematic", "collab", "equity", "top", "lotka",
    "bradford"),
  ...
)

Arguments

Value

A ggplot object.

See Also

Other plots: sm_palette_qualitative(), sm_plot_bradford(), sm_plot_citation_network(), sm_plot_collab(), sm_plot_equity_dashboard(), sm_plot_evolution(), sm_plot_heaps(), sm_plot_landscape(), sm_plot_lotka(), sm_plot_production(), sm_plot_thematic_map(), sm_plot_top(), sm_scale_color(), sm_theme()

Examples

corpus <- sm_example_corpus()
ggplot2::autoplot(corpus, type = "production")

Description

Test if an object is an sm_corpus

Usage

is_sm_corpus(x)

Arguments

Value

TRUE if x is an sm_corpus, FALSE otherwise.

See Also

Other corpus: as_sm_corpus(), sm_bind_corpora(), sm_build_corpus(), sm_concept_vocabularies(), sm_corpus(), sm_corpus_from_tables(), sm_dedupe(), sm_link_references(), sm_materialise(), sm_oa_statuses(), sm_save_corpus(), sm_validate(), validate_sm_corpus()

Examples

corpus <- sm_example_corpus()
is_sm_corpus(corpus)

Description

Test if an object is an sm_question

Usage

is_sm_question(x)

Arguments

Value

Logical.

See Also

Other question: sm_corpus_for_question(), sm_question(), sm_screen_against_question(), sm_screen_regex(), sm_screen_summary()

Description

Test if an object is an sm_trajectory

Usage

is_sm_trajectory(x)

Arguments

Value

Logical.

See Also

Other trajectory: sm_author_trajectory(), sm_plot_collab_turnover(), sm_plot_topic_pivots(), sm_plot_trajectory(), sm_researcher_profile(), sm_trajectory

Examples

is_sm_trajectory(1)

Description

Runs all available equity audits (geographic, gender, funding, OA status) on a corpus and returns a combined summary. Individual audit results can be passed as ... arguments to avoid re-computation; any missing audits are run with default parameters, except that the gender audit uses method = "manual" and audits that send web requests run only when online = TRUE.

Usage

## S3 method for class 'sm_audit_summary'
print(x, ...)

sm_audit_summary(corpus, ..., online = FALSE, call = rlang::caller_env())

Arguments

Value

An sm_audit_summary S3 object containing:

geographic

An sm_audit_geographic result.

gender

An sm_audit_gender result.

funding

An sm_audit_funding result. A funding audit that was not run has no funders, source = "none" and coverage = NA.

oa

An sm_audit_oa result.

overview

A tibble with one row per audit: audit, coverage (NA for an audit that was not run) and top_finding.

See Also

Other audit: sm_audit_funding(), sm_audit_gender(), sm_audit_geographic(), sm_audit_oa()

Examples

corpus <- sm_example_corpus(n_works = 50, n_authors = 20)
# The funding audit sends batched Crossref requests, so it is not run
# unless online = TRUE.
summary_audit <- sm_audit_summary(corpus)
summary_audit$overview

# A precomputed audit is used as it is. This funding audit sends no
# request because its copy of the corpus has no DOIs.
no_doi <- corpus
no_doi$works$doi <- NA_character_
funding <- sm_audit_funding(no_doi)
summary_audit <- sm_audit_summary(corpus, funding)
print(summary_audit)

Description

A small, documented, user-overridable dictionary of institution name variants used by sm_affiliation_match(). It covers common military / medical institution variants (including German synonyms) plus email-domain fallbacks, and is designed to be extended by appending rows.

Usage

sm_affiliation_dict

Format

A tibble with three columns:

institution

Canonical institution name (character).

pattern

A case-insensitive Perl-compatible regular expression matched against affiliation strings (character).

email_domain

Optional email domain used by the email-domain fallback, or NA (character).

Details

Patterns are case-insensitive PCRE with Unicode-aware ⁠\w⁠, ⁠\b⁠, ⁠\d⁠ and ⁠\s⁠: sm_affiliation_match() matches them in Unicode mode by default, so a row added for a German, French or Scandinavian name can use these classes next to non-ASCII letters (see the Details of sm_affiliation_match(), including its unicode argument). In that mode the shipped row ⁠\bnato\b⁠ no longer matches "NATO" directly followed by an umlaut, as in a German compound word. The first pattern that matches wins, in row order, so add specific rows before generic ones.

The rows for "Bundeswehr Hospital" match its German name, its abbreviation, or "hospital" after "Bundeswehr" or after "German Armed Forces", "German Army" or "German Military", with at most one word in between. A bare "military hospital", "army hospital" or "armed forces hospital", which names hospitals of many countries, stays unmatched; up to scimapR 0.4.0 these phrases matched "Bundeswehr Hospital" wherever they appeared. The dictionary is a starting point: for an audit of your own institutions, pass a dictionary of them to sm_affiliation_match().

See Also

sm_affiliation_match(), sm_affiliation_unmatched()

Examples

sm_affiliation_dict

Description

A tested, extensible institution matcher for the authorships table. It tags each authorship with the institution it belongs to, using a dictionary of name variants (multilingual and synonym-aware) and an optional email-domain fallback for records whose affiliation string is missing.

Because the matcher operates per authorship row, it naturally handles secondary / multiple affiliations per author (each authorship row is matched independently).

Usage

sm_affiliation_match(
  corpus,
  patterns = sm_affiliation_dict,
  fields = NULL,
  email_domain_fallback = TRUE,
  postcode_signal = FALSE,
  unicode = TRUE,
  call = rlang::caller_env()
)

Arguments

Details

Dictionary patterns are case-insensitive Perl-compatible regular expressions (PCRE) with Unicode-aware ⁠\w⁠, ⁠\b⁠, ⁠\d⁠ and ⁠\s⁠, matched against the affiliation text of each authorship. The rows are tried in dictionary order and the first pattern that matches wins, so list specific patterns before generic ones. sm_affiliation_unmatched() lists the most frequent affiliation strings that nothing matched.

In Unicode mode (unicode = TRUE, the default) each pattern is prefixed with the PCRE item ⁠(*UCP)⁠ unless its leading items already include it. Umlauts, accented letters and the sharp s are then word characters: ⁠\w⁠ consumes them, ⁠\b⁠ finds the edge of a word that starts or ends with one, and POSIX classes such as ⁠[[:alpha:]]⁠ are Unicode-aware too. In ASCII mode (unicode = FALSE) every non-ASCII letter is a non-word character, like punctuation: a pattern such as ⁠sanit\w+wesens⁠ cannot match the German word spelled with an a-umlaut, and ⁠\b⁠ finds no edge between a space and a word that starts with an umlaut. Unicode mode can therefore also lose matches. A pattern that used ⁠\W⁠, ⁠\S⁠, ⁠\D⁠ or ⁠\B⁠ to stand for a non-ASCII letter no longer matches it, and ⁠\b⁠ no longer finds an edge between an ASCII letter and a non-ASCII letter next to it: the shipped pattern ⁠\bnato\b⁠ does not match "NATO" directly followed by an umlaut, as in a German compound word. To keep ASCII semantics in one place, write the class out, for example ⁠[A-Za-z0-9_]⁠ instead of ⁠\w⁠.

unicode = FALSE gives the matches of scimapR 0.4.0. sm_attribute_institution() calls sm_affiliation_match() with its defaults, and so matches in Unicode mode, when the corpus has no institution_match column; running sm_affiliation_match(unicode = FALSE) first gives its earlier results. sm_count(level = "institution") no longer matches affiliations itself: it counts each authorship by its institution label, institution_match first, and by its raw affiliation string when it has none (see sm_count()).

The optional postcode column (used with postcode_signal = TRUE) is matched case-sensitively as a whole word, in the same mode: it is wrapped in ⁠\b⁠, any run of whitespace in it matches any amount of whitespace (including none), and it is used as a regular expression without escaping.

To extend the dictionary, append rows to sm_affiliation_dict (or build your own data frame with the same columns) and pass it as patterns. For example rbind(sm_affiliation_dict, tibble::tibble(institution = "My Uni", pattern = "my university", email_domain = "myuni.edu")).

Value

The corpus with its authorships table gaining (or having updated) four columns:

institution_match

Canonical institution name, or NA.

match_method

"pattern", "email_domain", "postcode", or "none".

match_signal

The signal that fired: "name_token", "email_domain", "postcode", or "none". Precedence (highest first): name token, email domain, postcode.

match_evidence

The actual substring / domain / postcode that triggered the match (an audit trail), or NA.

Type-stable: a corpus with no authorships is returned unchanged with the four columns present and 0 rows. See sm_affiliation_summary() for a tidy breakdown.

See Also

sm_attribute_institution(), sm_affiliation_dict, sm_affiliation_unmatched()

Other affiliation: sm_affiliation_methods(), sm_affiliation_signals(), sm_affiliation_summary(), sm_affiliation_unmatched(), sm_attribute_institution()

Examples

corpus <- sm_example_corpus(n_works = 5, n_authors = 5)
corpus$authorships$raw_affiliation[1] <- "Bundeswehrkrankenhaus Berlin"
matched <- sm_affiliation_match(corpus)
matched$authorships$institution_match[1]

# In Unicode mode \w consumes the umlaut of a German unit name
corpus$authorships$raw_affiliation[2] <-
  "Zentrales Institut des Sanit\u00e4tswesens Nordstadt"
dict <- list("Central Medical Institute" = "sanit\\w+wesens")
matched <- sm_affiliation_match(corpus, patterns = dict)
matched$authorships$institution_match[2]
ascii <- sm_affiliation_match(corpus, patterns = dict, unicode = FALSE)
ascii$authorships$institution_match[2]

Description

The set of values the match_method column of sm_affiliation_match() can take.

Usage

sm_affiliation_methods(describe = FALSE)

Arguments

Value

A character vector or a tibble.

See Also

sm_affiliation_signals()

Other affiliation: sm_affiliation_match(), sm_affiliation_signals(), sm_affiliation_summary(), sm_affiliation_unmatched(), sm_attribute_institution()

Examples

sm_affiliation_methods()

Description

The ordered set of values the match_signal column of sm_affiliation_match() / sm_affiliation_summary() can take. Using this helper (rather than hard-coding strings) means downstream filtering — e.g. separating institution-name matches from email-domain matches — cannot drift if the levels change.

Usage

sm_affiliation_signals(describe = FALSE)

Arguments

Value

A character vector (ordered, highest matching priority first) or a tibble of level/description.

See Also

sm_affiliation_match(), sm_affiliation_methods()

Other affiliation: sm_affiliation_match(), sm_affiliation_methods(), sm_affiliation_summary(), sm_affiliation_unmatched(), sm_attribute_institution()

Examples

sm_affiliation_signals()
sm_affiliation_signals(describe = TRUE)

Description

Tidy summary of the matches produced by sm_affiliation_match(), mirroring the audit-style summaries elsewhere in the package: counts of works and authorships flagged, broken down by institution and by match_signal.

Usage

sm_affiliation_summary(corpus, call = rlang::caller_env())

Arguments

Value

A tibble with columns institution, match_signal (a factor; see sm_affiliation_signals()), n_authorships, n_works, and example_evidence (a representative matched-evidence string for that institution x signal — the audit trail), sorted by n_authorships descending. Type-stable: a 0-row tibble (with a warning) when no matches are present.

See Also

sm_affiliation_match()

Other affiliation: sm_affiliation_match(), sm_affiliation_methods(), sm_affiliation_signals(), sm_affiliation_unmatched(), sm_attribute_institution()

Examples

corpus <- sm_example_corpus(n_works = 5, n_authors = 5)
corpus$authorships$raw_affiliation[1] <- "Bundeswehrkrankenhaus Berlin"
corpus <- sm_affiliation_match(corpus)
sm_affiliation_summary(corpus)

Description

Audits an affiliation dictionary. After sm_affiliation_match(), it lists the most frequent raw_affiliation strings of the authorships that no signal (name pattern, email domain or postcode) assigned to an institution, so the dictionary can be extended where that matters most.

Usage

sm_affiliation_unmatched(corpus, n = 25L, call = rlang::caller_env())

Arguments

Value

A tibble with columns raw_affiliation (character), n_authorships (integer: unmatched authorships with that string) and n_works (integer: distinct works among them), at most n rows, sorted by n_authorships descending and then by raw_affiliation in code-point order. Authorships whose raw_affiliation is NA or blank are not listed. Type-stable: a 0-row tibble when every affiliation matched, and a 0-row tibble with a warning when the corpus has not been matched.

See Also

sm_affiliation_match(), sm_affiliation_summary()

Other affiliation: sm_affiliation_match(), sm_affiliation_methods(), sm_affiliation_signals(), sm_affiliation_summary(), sm_attribute_institution()

Examples

fix <- readRDS(sm_example_files("example_realuse_corpus.rds"))
dict <- readr::read_csv(
  sm_example_files("example_realuse_affiliation_dict.csv"),
  col_types = "ccc"
)
matched <- sm_affiliation_match(fix, dict)
sm_affiliation_unmatched(matched, n = 5)

Description

Rolls institution matches (from sm_affiliation_match()) up to a controlled vocabulary, writing a normalised institution_id and institution_name onto the authorships table. Supports a ROR-backed vocabulary (via a user-supplied offline ROR table) or a "custom" vocabulary derived directly from the matched institution names.

Usage

sm_attribute_institution(
  corpus,
  vocabulary = c("ror", "custom"),
  ror_table = NULL,
  call = rlang::caller_env()
)

Arguments

Details

The institution_match column stays as it is. sm_count(level = "institution") labels each authorship by institution_match before institution_name and institution_id, so the roll-up to the vocabulary reaches the counts only for authorships without a match, and the "affiliation" breakdown of sm_coverage_audit() uses institution_name only when no authorship has a match. Up to scimapR 0.4.0, sm_count() counted by institution_id whenever any authorship had one. To count or break down by the vocabulary, put its names first, for example:

a <- corpus$authorships
corpus$authorships$institution_match <-
  dplyr::coalesce(a$institution_name, a$institution_match)

The ids written to matched rows are ROR ids or ⁠CUST:⁠ ids, which join corpus$institutions only where that table lists them, so functions that read the institutions table for these rows (such as sm_audit_geographic(by = "region")) find no entry for them. Unmatched rows keep their earlier ids and so their entries. Up to scimapR 0.4.0, unmatched rows lost their institution_id.

Value

The corpus with institution_id and institution_name set on the authorships rows that match the vocabulary (for "ror", institution_id holds the ROR id; for "custom", ⁠CUST:⁠ and the matched name). Unmatched rows keep the institution_id and institution_name they had, such as the OpenAlex institution ids of a fetched corpus, or NA – the function never errors on unmatched affiliations. Type-stable.

See Also

sm_affiliation_match()

Other affiliation: sm_affiliation_match(), sm_affiliation_methods(), sm_affiliation_signals(), sm_affiliation_summary(), sm_affiliation_unmatched()

Examples

ror <- utils::read.csv(
  system.file("extdata", "example_ror.csv", package = "scimapR"),
  stringsAsFactors = FALSE
)
corpus <- sm_example_corpus(n_works = 5, n_authors = 5)
corpus$authorships$raw_affiliation[1] <- "Charite Universitatsmedizin Berlin"
corpus <- sm_affiliation_match(corpus)
corpus <- sm_attribute_institution(corpus, vocabulary = "ror",
                                   ror_table = ror)
corpus$authorships$institution_name[1]

Description

Audits the distribution of funding sources across works in a corpus. Funding metadata is extracted from either Crossref funder fields or OpenAlex funders and awards fields (also accepting legacy grants), depending on the source parameter.

The result includes funder counts, the proportion of works with known funding, and a concentration analysis.

Usage

sm_audit_funding(
  corpus,
  source = c("crossref", "openalex"),
  api_key = Sys.getenv("OPENALEX_API_KEY"),
  mailto = Sys.getenv("SCIMAPR_MAILTO"),
  call = rlang::caller_env()
)

## S3 method for class 'sm_audit_funding'
print(x, ...)

Arguments

Value

An sm_audit_funding S3 object containing:

funders

Tibble with columns funder_name, funder_doi, n_works, pct (percentage of checked works).

coverage

Proportion of checked works with at least one funder; NA if none could be checked, and 0 for an empty corpus.

n_works_total

Total number of works in corpus.

n_works_funded

Number of works with funding data.

n_works_checked

Works returned by the source.

n_works_failed

Works whose request failed; reported in a warning.

n_works_no_id

Works without an identifier for the chosen source.

source

The data source used.

See Also

Other audit: print.sm_audit_summary(), sm_audit_gender(), sm_audit_geographic(), sm_audit_oa()

Examples

# Works without a DOI need no Crossref lookup, so this runs offline and
# returns an empty audit.
corpus <- sm_example_corpus(n_works = 20, n_authors = 10)
corpus$works$doi <- NA_character_
funding <- sm_audit_funding(corpus, source = "crossref")
print(funding)

# Live lookup: batched Crossref requests (the example DOIs are invented,
# so no funders are found).
corpus <- sm_example_corpus(n_works = 3, n_authors = 3)
funding <- sm_audit_funding(corpus, source = "crossref")
print(funding)

Description

Audits the inferred gender distribution of authors in a corpus. Gender is inferred from first names using the specified method. Results include the overall distribution, authorship-position breakdown, and a coverage metric.

This function infers a binary gender proxy from names, which has well-documented limitations. See the Limitations section in the printed output and the details below.

Usage

sm_audit_gender(
  corpus,
  method = c("genderize", "ssa", "manual"),
  api_key = Sys.getenv("GENDERIZE_API_KEY"),
  cache_dir = tools::R_user_dir("scimapR", "cache"),
  call = rlang::caller_env()
)

## S3 method for class 'sm_audit_gender'
print(x, ...)

Arguments

Details

Known limitations of automated gender inference:

• Name-based methods assign a binary gender proxy, which does not capture the full spectrum of gender identity.

• Accuracy varies dramatically by cultural context: names from East Asian, South Asian, and many African cultures are poorly served by Western-trained models.

• Non-binary, transgender, and gender-diverse individuals are systematically misclassified.

• The method cannot account for name changes, pen names, or initialised first names.

Value

An sm_audit_gender S3 object containing:

distribution

Tibble with columns inferred_gender, count, pct.

by_position

Tibble breaking down gender by authorship position: position_type ("first", "corresponding" for a corresponding author who is not first, else "middle/last"), inferred_gender, count and pct. Each work and author counts once, however many address rows the author has; corresponding authors are those with authorships$is_corresponding = TRUE (NA counts as not corresponding).

coverage

Proportion of authors with an inferred gender.

method

The method used.

n_names_failed

Number of distinct names without an API answer; these are not cached and are retried on the next run.

confidence_summary

Summary statistics for gender confidence scores.

See Also

Other audit: print.sm_audit_summary(), sm_audit_funding(), sm_audit_geographic(), sm_audit_oa()

Examples

corpus <- sm_example_corpus()
# Using manual method (no API call needed for examples):
gender_audit <- sm_audit_gender(corpus, method = "manual")
print(gender_audit)

Description

Audits the geographic distribution of a corpus, tabulating the representation of countries, regions, or World Bank income tiers. The analysis can be weighted by work count, total citations, first-authorship, or corresponding authorship. Each work counts once per distinct known group, with optional fractional credit.

The result includes a Gini coefficient for concentration and a coverage metric (proportion of works with known geography).

Usage

sm_audit_geographic(
  corpus,
  by = c("country", "region", "income_tier"),
  weight = c("count", "citations", "first-author", "corresponding"),
  method = c("full", "fractional"),
  call = rlang::caller_env()
)

## S3 method for class 'sm_audit_geographic'
print(x, ...)

Arguments

Details

Coverage uses all corpus works as its denominator and counts works with at least one known group after the authorship filter. Missing geography is excluded from credit allocation; incomplete metadata can therefore change both coverage and the shares assigned to known groups.

Value

An sm_audit_geographic S3 object containing:

distribution

Tibble with columns group, count, citations, pct, credit and weighted_citations. Counts and raw citations include each work once per group under both methods. Percentages and the Gini use credit, or weighted citations for citation weighting. Count percentages describe work-group pairs under full counting and works under fractional counting.

gini

Gini coefficient of the distribution.

coverage

Proportion of works with at least one known geography.

by

The grouping variable used.

weight

The weighting method used.

method

The counting rule used.

x invisibly (print methods).

See Also

Other audit: print.sm_audit_summary(), sm_audit_funding(), sm_audit_gender(), sm_audit_oa()

Examples

corpus <- sm_example_corpus()
geo <- sm_audit_geographic(corpus)
print(geo)

Description

Audits the open-access (OA) status distribution of works in a corpus. Uses the statuses of sm_oa_statuses(); every recognised status except closed is open. Missing and unrecognised values are unknown. The open percentage uses all works as the denominator and agrees with sm_summary_works() to one decimal place.

The result includes counts and percentages for each OA category, as well as a coverage metric.

Usage

sm_audit_oa(corpus, call = rlang::caller_env())

## S3 method for class 'sm_audit_oa'
print(x, ...)

Arguments

Value

An sm_audit_oa S3 object containing:

distribution

Tibble with columns oa_status, count, pct, and logical is_open (NA for unknown statuses).

pct_open

Percentage of works with an open status in sm_oa_statuses().

coverage

Proportion of works with a known OA status.

n_works

Total number of works.

See Also

Other audit: print.sm_audit_summary(), sm_audit_funding(), sm_audit_gender(), sm_audit_geographic()

Examples

corpus <- sm_example_corpus()
oa <- sm_audit_oa(corpus)
print(oa)

Description

Analyses the career trajectory of a single author within a corpus, partitioning their publication record into n_periods career stages. For each period, the function computes:

• Career stages: dominant topics (the three concept names found in most of the period's works) and publication volume.

• Topic pivots: how much the research focus shifted between periods, measured as the Jaccard distance of concept sets.

• Collaborator turnover: Jaccard similarity of co-author sets across consecutive periods, plus counts of new, kept, and lost collaborators.

• Emerging collaborators: co-authors who first appear in the most recent period.

• H-index curve: cumulative h-index at the end of each period.

• Novelty curve: average concept novelty per period, the mean number of distinct concept names per work (over the works with concepts).

• Citation acceleration: mean citations per period relative to the corpus-wide mean for the same years.

Usage

sm_author_trajectory(
  corpus,
  orcid = NULL,
  author_id = NULL,
  n_periods = 5L,
  call = rlang::caller_env()
)

## S3 method for class 'sm_trajectory'
print(x, ...)

## S3 method for class 'sm_trajectory'
format(x, ...)

Arguments

Value

An sm_trajectory S3 object.

• print(): x invisibly.

• format(): A single character string.

See Also

Other trajectory: is_sm_trajectory(), sm_plot_collab_turnover(), sm_plot_topic_pivots(), sm_plot_trajectory(), sm_researcher_profile(), sm_trajectory

Examples

corpus <- sm_example_corpus()
# Use the first author's ID
traj <- sm_author_trajectory(corpus, author_id = "A000000001")
print(traj)

Description

Combine two corpora, assigning new IDs to colliding records in the second input and rewriting their links in every table and embedding. This includes yearly citation counts (citations_by_year). The bound corpus keeps the metadata of corpus1, with the local input files (sm_register_input()) of both corpora, each file and content once.

Usage

sm_bind_corpora(corpus1, corpus2, mix_models = FALSE, verbose = TRUE)

Arguments

Details

Work IDs shared by the inputs always get new values in corpus2. Author, source and institution IDs are shared only when their lookup rows agree on every common column, including missing and list values. Other collisions get new IDs above the numbers used by either input; external cited work IDs are reserved too. Shared DOIs remain separate records until sm_dedupe() is called. The merged corpus is unlocked.

Embeddings are bound when both corpora have them, with the same number of dimensions, and when they come from the same model as metadata$embedding records it (see sm_corpus()); the bound corpus keeps that record. Embeddings that record different models, or of which only one corpus records its model, are not comparable: the function stops (class scimapR_embedding_model_mismatch) unless mix_models = TRUE, which binds them and records the model as "mixed". Embeddings of two corpora that record no model, such as corpora embedded by earlier versions of scimapR, are bound as before, and the model of the bound rows stays unrecorded. When only one corpus has embeddings, or their widths differ, the bound corpus has none and warns with class scimapR_bind_embeddings_dropped.

Value

An sm_corpus object.

See Also

Other corpus: as_sm_corpus(), is_sm_corpus(), sm_build_corpus(), sm_concept_vocabularies(), sm_corpus(), sm_corpus_from_tables(), sm_dedupe(), sm_link_references(), sm_materialise(), sm_oa_statuses(), sm_save_corpus(), sm_validate(), validate_sm_corpus()

Description

Combine multiple file reads and/or API fetches into a single deduplicated sm_corpus.

Usage

sm_build_corpus(..., dedupe = TRUE, mix_models = FALSE, verbose = TRUE)

Arguments

Details

The inputs are bound in turn with sm_bind_corpora(), so its rules for embeddings apply: embeddings that record different models in metadata$embedding, or of which only one input records its model, stop the function (class scimapR_embedding_model_mismatch) unless mix_models = TRUE. Pass every file of a split export; the IDs stay unique.

Value

An sm_corpus object.

See Also

Other corpus: as_sm_corpus(), is_sm_corpus(), sm_bind_corpora(), sm_concept_vocabularies(), sm_corpus(), sm_corpus_from_tables(), sm_dedupe(), sm_link_references(), sm_materialise(), sm_oa_statuses(), sm_save_corpus(), sm_validate(), validate_sm_corpus()

Examples

c1 <- sm_example_corpus(n_works = 50, seed = 1)
c2 <- sm_example_corpus(n_works = 50, seed = 2)
combined <- sm_build_corpus(c1, c2, dedupe = TRUE)

Description

A corpus certificate is a self-contained YAML document that records how a corpus was made and lets others check it: the corpus hash (sm_hash_corpus()) and the size of its tables, the steps recorded in its provenance (searches, file reads, enrichments and refreshes), a summary of its screening decisions, its embeddings and their model, and the local input files registered with sm_register_input(), each with its SHA-256 hash.

sm_certificate() creates a certificate from a corpus. sm_verify_certificate() compares a live corpus, and its local input files, against a certificate to detect divergence. sm_rebuild_from_cert() runs the recorded searches again (see "Rebuilding a corpus"); file reads and local inputs cannot be replayed, and verification checks them instead.

Each recorded step (queries) has a source, query, engine and kind: "api_query" for searches of a bibliographic API, "file" for records read from a local file (the file relative to the working directory when it lies below it, else its name), "enrichment" for steps such as sm_enrich_openalex_impact() that added fields to works already in the corpus, "refresh" for sm_refresh() and "manual" for synthetic records. The searches of sm_fetch_openalex(), sm_fetch_pubmed(), sm_fetch_crossref() and sm_corpus_for_question() also record their filter and n_max. The kind comes from the provenance column kind, else (for corpora and certificates written before kinds were recorded) from the source label, the query and the engine (see sm_provenance()).

Usage

sm_certificate(corpus, path = NULL, call = rlang::caller_env())

sm_rebuild_from_cert(cert, verbose = TRUE, call = rlang::caller_env())

sm_verify_certificate(
  corpus,
  cert,
  base_dir = NULL,
  call = rlang::caller_env()
)

## S3 method for class 'sm_certificate'
print(x, ...)

## S3 method for class 'sm_cert_verification'
print(x, ...)

Arguments

Details

Certificates write their timestamps (created, the corpus's build_date and last_refresh, and the registered_at of each local input) in UTC as ISO 8601 text, for example ⁠2026-10-17T11:45:00Z⁠, and read back both this form and the older one with a numeric offset. Printing a certificate shows created in UTC as well, so a certificate read back from YAML prints the same time as the one it was written from. The corpus timestamps may be stored as POSIXct, a Date, numeric seconds since 1970-01-01 UTC (as in corpora restored from tables) or ISO 8601 text. A missing or implausible value is written as NA.

embedding_info describes corpus$embeddings: its size (n_works, n_dimensions) and, from the record in corpus$metadata$embedding (see sm_corpus()), the model behind it: model, model_id, base_model_id, revision, base_revision, text, prefix, models, versions and created. method names how the embeddings were made: "sentence-transformers" or "adapters" (sm_embed_works()), "semantic-scholar" (sm_enrich_specter(), sm_fetch_semantic_scholar()) or "mixed" for rows of several models; it is "unknown" for embeddings without a record that fits them (such as embeddings made by earlier versions of scimapR, or a matrix replaced by hand) and "none" when the corpus has no embeddings. In these two cases model and model_id say the same.

Value

For sm_certificate(): an sm_certificate S3 object (list), also written to path if provided. For sm_rebuild_from_cert(): an sm_corpus with the works of the replayed searches. For sm_verify_certificate(): an sm_cert_verification S3 object, a list with pass, checks, n_not_checked, corpus_hash (computed with the certificate's hash version), cert_hash, cert_created and hash_version.

x invisibly (print methods).

Local inputs

The local_inputs block lists the files of metadata$local_inputs (sm_register_input()) with their path, path_abs, root, basename, sha256, bytes, role, builder and registered_at. It is a tibble in the certificate object, with no rows when no file is registered, and a list with one entry per file in the YAML file.

sm_verify_certificate() adds one check per file, named ⁠input:<basename>⁠ (⁠input:<path>⁠ when files share a name). It looks for a file with a relative path in base_dir, then in the folder of the certificate file (when cert is a path), then in the working directory, and last at path_abs; a file with an absolute path is looked for there and at path_abs. The first file found is hashed again, its check passes when the hash is the recorded one, and the check's location says where it was found. A file found nowhere is not checked, and the function warns (class scimapR_cert_inputs_missing), naming the files and the folders searched, because a verification run from the wrong folder would otherwise pass without a word.

Verification results

checks has one row per check, with the columns check, expected, actual (for a local input the first 16 characters of the hashes, or "missing"), pass, status and location. status (character) is "pass", "fail" or "not checked"; pass is TRUE, FALSE, or NA for a check that was not made. The overall pass is TRUE unless a check failed, so it can be TRUE while n_not_checked counts checks that were not made. The corpus hash is computed with the hash_version the certificate records, or with version 1 for certificates written before scimapR 0.5.0 (see sm_hash_corpus()).

Rebuilding a corpus

sm_rebuild_from_cert() replays the "api_query" steps of OpenAlex, PubMed and Crossref with the fetcher of their source (sm_fetch_openalex(), sm_fetch_pubmed() or sm_fetch_crossref()), the engine they were made with and the recorded filter and n_max. The searches of sm_corpus_for_question() record the client package that ran them (openalexR, rentrez or rcrossref) as their engine. A step without n_max, as in certificates of earlier versions, fetches up to the number of works of the certified corpus, with a message. The other steps (file reads, enrichments, refreshes, synthetic records and searches of other sources) and the local inputs cannot be replayed, and a message of class scimapR_cert_not_replayed names them. When a certificate has no search to replay but records file reads or local inputs, the function stops instead, with an error of class scimapR_cert_not_replayable that lists them, with the role, hash and builder of each local input. Without a search to replay, file reads or local inputs (only synthetic records, enrichments, refreshes or searches of other sources, or no step at all), it returns an empty corpus.

The rebuilt corpus holds the works of the replayed searches, without duplicate DOIs and with new work ids, and their provenance; its other tables are empty. It shows what the recorded searches find today, so it does not pass sm_verify_certificate(): its hash and table sizes differ from those of the certified corpus.

See Also

Other reproducibility: sm_cite_corpus(), sm_diff_corpora(), sm_hash_corpus(), sm_provenance(), sm_register_input(), sm_snapshot()

Examples

corpus <- sm_example_corpus()
tracker <- tempfile(fileext = ".csv")
writeLines(c("doi,unit", "10.9999/example.1,Unit A"), tracker)
corpus <- sm_register_input(corpus, tracker, role = "publication tracker")
cert <- sm_certificate(corpus)
print(cert)
verification <- sm_verify_certificate(corpus, cert)
print(verification)

# a changed input file fails its check
cat("10.9999/example.2,Unit B\n", file = tracker, append = TRUE)
sm_verify_certificate(corpus, cert)$pass

Description

Ask a question of a corpus using retrieval-augmented generation (RAG). The function retrieves the most relevant works from the corpus using embeddings, TF-IDF, or a hybrid method, then sends them as context to a chat provider along with the user's question. The response includes inline citations keyed to work IDs. No hallucinated references are possible because citations are constrained to retrieved works.

Usage

sm_chat(
  corpus,
  question,
  provider = NULL,
  retrieve_n = 30L,
  retrieve_method = c("embedding", "tfidf", "hybrid"),
  cite = TRUE,
  max_tokens = lifecycle::deprecated(),
  temperature = lifecycle::deprecated(),
  call = rlang::caller_env()
)

## S3 method for class 'sm_chat_response'
print(x, ...)

Arguments

Value

An sm_chat_response S3 object with fields:

answer

Character. The chat provider's response.

citations

Tibble of cited work IDs and their details.

retrieved_works

Tibble of the works sent as context.

prompt_hash

SHA-256 hash of the full prompt for reproducibility.

model

Character. The model used.

timestamp

POSIXct. When the response was generated.

question

The original question.

retrieve_method

The retrieval method used.

See Also

Other chat: sm_chat_render()

Examples

## Not run: 
# Requires ellmer package and chat-provider API access:
# response <- sm_chat(corpus, "What methods dominate?",
#                     provider = ellmer::chat_openai())

## End(Not run)

Description

Converts an sm_chat_response object into a formatted document string. The rendered output includes the question, answer with inline citations, and a reference list. Supported formats are Markdown, HTML, and Quarto.

Usage

sm_chat_render(
  response,
  format = c("markdown", "html", "quarto"),
  call = rlang::caller_env()
)

Arguments

Value

A character string containing the rendered response. Printed to the console and returned invisibly.

See Also

Other chat: sm_chat()

Examples

# Requires a chat response object:
# rendered <- sm_chat_render(response, format = "markdown")
# cat(rendered)

Description

Adds citation-maturity flags to the works table so that downstream trend and impact functions can shade or exclude recent, citation-immature years instead of treating their (artificially low) citation counts as final.

Works published within the most recent lag years – relative to the corpus's "as-of" date (metadata$last_refresh if present, else the maximum publication year) – are flagged provisional. last_refresh may be a POSIXct, a Date, numeric seconds since 1970-01-01 UTC (as in corpora restored from tables) or ISO 8601 text; its year is taken in UTC. A value that is not a plausible date (for example milliseconds stored as seconds) is ignored with a message.

Usage

sm_citation_maturity(corpus, lag = 2L, call = rlang::caller_env())

Arguments

Value

The corpus with two logical columns added to works: citation_mature (TRUE when the year is mature) and cnci_provisional (the negation; TRUE when citation-based indicators should be treated as provisional). Type-stable: an empty corpus gains the columns with 0 rows.

See Also

sm_its(), sm_metric_fnci()

Other counting: sm_citation_window(), sm_count(), sm_metric_summary()

Examples

corpus <- sm_example_corpus(n_works = 20, seed = 1)
corpus <- sm_citation_maturity(corpus, lag = 2)
table(corpus$works$cnci_provisional)

Description

Counts each work's citations in its publication year and the window years after it, from the yearly citation counts of the corpus (citations_by_year, written by the OpenAlex readers and sm_enrich_openalex_impact()), and says whether the counts cover the whole window. Fixed windows compare old and recent works fairly, which cited_by_count (all citations up to now) does not.

Usage

sm_citation_window(corpus, window = 3L, call = rlang::caller_env())

Arguments

Details

OpenAlex gives the citations of the ten most recent years only (its counts_by_year: the year of the counts and the nine before it), and leaves out the years without citations. A year missing from citations_by_year therefore counts 0 for a work that has yearly counts. The date of the counts is works$citations_as_of (the day OpenAlex last updated the record, or of the fetch); its year is still running, so it counts as partial. A corpus restored with sm_corpus_from_tables() may hold this date as a day count (days since 1970-01-01, as SQLite keeps dates) or as ISO 8601 text; both are read as the date they stand for.

A window is complete when the work has a publication year and yearly counts with a known date, its first year (the publication year) is no earlier than nine years before the year of the counts, and its last year is before the year of the counts. Otherwise complete is FALSE and citations is a lower bound: the citations the counts hold for the window years. A work without a publication year, or without yearly counts (no rows in citations_by_year and no citations_as_of), gets citations = NA.

The result depends only on the corpus, never on today's date, so it is the same in every session. A corpus without citations_by_year gives NA for every work, with one message (class scimapR_no_citations_by_year) that points to sm_enrich_openalex_impact().

Value

A tibble with one row per work, in the order of corpus$works: work_id (character), year (integer, the publication year), window (integer), citations (integer) and complete (logical). A corpus without works gives a 0-row tibble with these columns.

See Also

sm_citation_maturity(), sm_enrich_openalex_impact()

Other counting: sm_citation_maturity(), sm_count(), sm_metric_summary()

Examples

works <- data.frame(
  work_id = c("W1", "W2", "W3"),
  title = c("Early work", "Recent work", "Unknown counts"),
  year = c(2018L, 2024L, 2019L),
  citations_as_of = as.Date(c("2026-10-17", "2026-10-17", NA))
)
yearly <- data.frame(
  work_id = c("W1", "W1", "W1", "W2"),
  year = c(2018L, 2019L, 2021L, 2025L),
  cited_by_count = c(2L, 5L, 4L, 3L)
)
corpus <- sm_corpus_from_tables(
  list(works = works, citations_by_year = yearly)
)
sm_citation_window(corpus, window = 3)

Description

Produces a structured citation block for a corpus that can be included in manuscripts. The block documents the number of works, sources queried, date range, scimapR version, and corpus hash for reproducibility.

The build date is the UTC date of metadata$build_date, which may be a POSIXct, a Date, numeric seconds since 1970-01-01 UTC or ISO 8601 text; a missing or implausible value is shown as unknown.

Usage

sm_cite_corpus(
  corpus,
  style = c("text", "bibtex", "yaml"),
  call = rlang::caller_env()
)

Arguments

Value

A character string containing the formatted citation. Printed to the console and returned invisibly.

See Also

Other reproducibility: sm_certificate(), sm_diff_corpora(), sm_hash_corpus(), sm_provenance(), sm_register_input(), sm_snapshot()

Examples

corpus <- sm_example_corpus()
sm_cite_corpus(corpus)

Description

Analyses how research clusters evolve across time windows. For each pair of consecutive windows, clusters are linked either by the overlap of the terms in their titles (link_by = "terms", the default) or, after one clustering of the whole corpus, by their shared cluster_id (link_by = "cluster_id").

Usage

sm_cluster_evolution(
  corpus,
  time_windows = NULL,
  link_threshold = 0.3,
  link_by = c("terms", "cluster_id"),
  noise = 0L,
  call = rlang::caller_env()
)

Arguments

Details

The function:

1. Splits works into time windows.

2. Within each window, identifies the cluster composition.

3. Between consecutive windows, links clusters. A work sits in exactly one window, so clusters cannot share works; "terms" therefore compares the terms of their titles and links the pairs whose Jaccard similarity reaches link_threshold, while "cluster_id" links each cluster to itself and reports the title-term Jaccard of its two windows without filtering.

With one clustering of the whole corpus, the same cluster_id in two windows is the continuity, and large clusters rarely reach the term threshold, so "terms" can report no transition at all. Use link_by = "cluster_id" there.

Value

A list with two components and the attribute link_by (the linking mode used):

snapshots

A tibble with columns window, cluster_id, n_works, top_terms (a character string of representative terms) and share (the cluster's share of the window's clustered works, noise included, so the shares of a window sum to 1).

transitions

A tibble with columns from_window, to_window, from_cluster, to_cluster, jaccard (the Jaccard similarity of the two clusters' title terms, rounded to 4 digits; 0 when neither has any) and n_shared (the number of shared title terms).

See Also

Other clustering: sm_cluster_hdbscan(), sm_cluster_kmeans(), sm_cluster_label(), sm_cluster_leiden()

Examples

corpus <- sm_example_corpus(with_embeddings = TRUE)
corpus <- sm_cluster_kmeans(corpus, k = 5, reducer = "pca")
evo <- sm_cluster_evolution(corpus)
evo$snapshots
evo$transitions

# one clustering for the whole corpus: follow each cluster id
evo <- sm_cluster_evolution(corpus, link_by = "cluster_id")
evo$transitions

Description

Clusters works using the HDBSCAN (Hierarchical Density-Based Spatial Clustering of Applications with Noise) algorithm via dbscan::hdbscan(). Optionally reduces embedding dimensions first with UMAP or PCA.

Usage

sm_cluster_hdbscan(
  corpus,
  min_cluster_size = 15L,
  min_samples = NULL,
  reducer = c("umap", "pca", "none"),
  n_components = 5L,
  call = rlang::caller_env()
)

Arguments

Details

Work IDs and embedding row names must be unique. Repeated IDs stop with class scimapR_duplicate_ids; use sm_validate() to inspect them.

Requires embeddings in corpus$embeddings. Compute them first with sm_embed_works() or load from cache with sm_embed_load().

The reducer step helps HDBSCAN work in a lower-dimensional space where density estimation is more reliable.

UMAP requires its optional dependency only when reduction is needed. If the PCA dependency raises the known NULL-input error on older R, reduction retries with base SVD initialisation and reports scimapR_umap_pca_fallback. Other errors propagate unchanged.

Value

The input corpus with a cluster_id column added to corpus$works. Noise points (not assigned to any cluster) receive cluster_id = 0L. Works without a row in corpus$embeddings (such as works without text, see sm_embed_works()) get NA.

See Also

Other clustering: sm_cluster_evolution(), sm_cluster_kmeans(), sm_cluster_label(), sm_cluster_leiden()

Examples

corpus <- sm_example_corpus(with_embeddings = TRUE)
corpus <- sm_cluster_hdbscan(corpus, min_cluster_size = 10L)
table(corpus$works$cluster_id)

Description

Clusters works using K-means clustering via stats::kmeans(). Optionally reduces embedding dimensions first with UMAP or PCA.

Usage

sm_cluster_kmeans(
  corpus,
  k,
  reducer = c("umap", "pca", "none"),
  n_components = 5L,
  call = rlang::caller_env()
)

Arguments

Details

Work IDs and embedding row names must be unique. Repeated IDs stop with class scimapR_duplicate_ids; use sm_validate() to inspect them.

Requires embeddings in corpus$embeddings. Compute them first with sm_embed_works() or load from cache with sm_embed_load().

K-means is deterministic given a fixed random seed. Consider setting a seed before calling this function for reproducibility.

UMAP requires its optional dependency only when reduction is needed. If the PCA dependency raises the known NULL-input error on older R, reduction retries with base SVD initialisation and reports scimapR_umap_pca_fallback. Other errors propagate unchanged.

Value

The input corpus with a cluster_id column added to corpus$works. Works without a row in corpus$embeddings (such as works without text, see sm_embed_works()) get NA.

See Also

Other clustering: sm_cluster_evolution(), sm_cluster_hdbscan(), sm_cluster_label(), sm_cluster_leiden()

Examples

corpus <- sm_example_corpus(with_embeddings = TRUE)
corpus <- sm_cluster_kmeans(corpus, k = 5, reducer = "pca")
table(corpus$works$cluster_id)

Description

Assigns human-readable labels to clusters by extracting representative terms. Supports TF-IDF-based extraction (default), labels from the author and indexer keywords of the corpus, YAKE keyword extraction, or labels from a chat provider via the ellmer package.

Usage

sm_cluster_label(
  corpus,
  method = c("tfidf", "keywords", "yake", "llm"),
  n_terms = 5L,
  llm_provider = NULL,
  min_works = NULL,
  ngram = 1:2,
  stop_terms = NULL,
  noise = 0L,
  seed = NULL,
  call = rlang::caller_env()
)

Arguments

Details

Noise and clusters without a label: works whose cluster_id is NA or noise get no label from any method, and the "tfidf" and "keywords" methods leave them out of the works that make a term common or rare in the corpus. A cluster whose works share no term, such as a cluster of a single work, gets no label either, with a message.

TF-IDF method: Splits the titles and abstracts of the works into terms of ngram words. A term's score in a cluster is the share of the cluster's works that contain it times log(N / n), where N is the number of clustered works and n the number of them that contain the term. As the score counts works, not occurrences, and a term must occur in at least min_works works of the cluster, a term that one paper repeats cannot name a cluster. The n_terms terms with the highest scores form the label; ties go to the term that occurs more often in the cluster, then to the first in alphabetical order. A longer term takes the place of a shorter one it contains when it occurs in at least 40% of the shorter term's works ("cancer" gives way to "testicular cancer", and to "metastatic testicular cancer" only if that also occurs in 40% of the works with "cancer"), and a term contained in one already chosen is skipped. When a term overlaps one already chosen, as "heat illness" overlaps "exertional heat", the phrase they form ("exertional heat illness") counts as a longer term that contains both, also beyond the lengths in ngram, if it occurs in enough of the cluster's works to be a term of the label; so one phrase does not fill two places of a label.

Words of fewer than 3 characters, numbers and stop words are left out, and no term spans one of them, the end of a sentence or clause, or the border between title and abstract. The stop words of every work are the English ones of tidytext::stop_words, plus the Snowball list of the stopwords package for the work's own language in works$language, given as a name such as "German", an ISO 639-1 code such as "de" or an ISO 639-2 code such as "ger". Works without a language get the English list only, and the lists of other languages are never pooled, so German stop words such as "war" or "die" stay in English text. Without the stopwords package, or for a language it has no list for, only the English list is used, with a message.

Singular and plural count as one term: a final "ies" becomes "y", and a final "s" is dropped after a letter other than s, i, u, o, a and y, so "fractures" counts with "fracture" and "ticks" with "tick", while "diagnosis", "virus", "class" and "days" stay as they are. A possessive "'s" is dropped as well, and words such as "species", "series", "diabetes", "measles" or "aids" are kept as written, as are names of fields such as "logistics", "genetics", "genomics" or "pediatrics", which would otherwise count with the adjectives "logistic", "genetic", "genomic" or "pediatric". A label shows each term in the spelling the cluster's works use most often.

Keyword method: The same scoring on the author keywords and the indexer keywords (such as Keywords Plus or MeSH) in corpus$concepts, the families "author_keyword" and "indexer_keyword" of sm_concept_vocabularies(). Keywords are compared ignoring case, with hyphens read as spaces and plurals folded as above, and containment is checked on whole words; keywords that overlap form no phrase. A work counts once for each keyword, however many of its vocabularies give it. A keyword is shown as an author keyword spells it, or else in lower case. A cluster without a keyword in at least 2 of its works gets the TF-IDF terms of its titles instead; when the clustered works have no such keywords at all, every cluster does, with a message.

YAKE method: Uses a simplified YAKE-like scoring based on term frequency, position, and spread across works within each cluster, with the stop words of the TF-IDF method.

LLM method: Sends titles and abstracts of each cluster to an LLM with a prompt asking for a concise topical label. Requires the ellmer package and a configured provider.

Value

The input corpus with two columns added to corpus$works, in place of earlier ones:

cluster_label

The terms of the work's cluster joined with "; ", the best first; NA for noise, for works without a cluster and for clusters that got no label.

cluster_label_basis

How the label was made: "tfidf" or "keywords" when every term occurs in at least min_works of the cluster's works; "tfidf (relaxed)" or "keywords (relaxed)" when no term reached min_works, so that the terms occur in at least 2 works; "tfidf (fallback)" or "tfidf (fallback, relaxed)" when method = "keywords" found no keyword in 2 or more of the cluster's works and took the terms of its titles; "yake" or "llm" for these methods; NA where cluster_label is NA.

See Also

Other clustering: sm_cluster_evolution(), sm_cluster_hdbscan(), sm_cluster_kmeans(), sm_cluster_leiden()

Examples

corpus <- sm_example_corpus(with_embeddings = TRUE)
corpus <- sm_cluster_kmeans(corpus, k = 5, reducer = "pca")
corpus <- sm_cluster_label(corpus, method = "tfidf", n_terms = 3L)
unique(corpus$works[, c("cluster_id", "cluster_label",
                        "cluster_label_basis")])

# keyword labels: the real-use example corpus grouped by journal category
fix <- readRDS(sm_example_files("example_realuse_corpus.rds"))
fix$works$cluster_id <- match(fix$works$field, unique(fix$works$field))
fix <- sm_cluster_label(fix, method = "keywords", n_terms = 3L)
unique(fix$works[, c("field", "cluster_label", "cluster_label_basis")])

Description

Clusters works using the Leiden community detection algorithm via igraph::cluster_leiden(). If no network is provided, a semantic similarity network is built automatically from embeddings.

Usage

sm_cluster_leiden(
  corpus,
  network = NULL,
  resolution = 1,
  call = rlang::caller_env()
)

Arguments

Details

Work IDs and network node names must be unique. Repeated IDs stop with class scimapR_duplicate_ids; use sm_validate() to inspect them.

The Leiden algorithm (Traag et al., 2019) is a refinement of the Louvain algorithm that guarantees well-connected communities. It operates on the edge weights of the network.

When network = NULL, embeddings must be present in the corpus so that sm_network_semantic() can build a k-NN graph.

Value

The input corpus with a cluster_id column added to corpus$works. Works that are not nodes of the network, such as works without a row in corpus$embeddings, get NA.

See Also

Other clustering: sm_cluster_evolution(), sm_cluster_hdbscan(), sm_cluster_kmeans(), sm_cluster_label()

Examples

corpus <- sm_example_corpus(with_embeddings = TRUE)
corpus <- sm_cluster_leiden(corpus, resolution = 1.0)
table(corpus$works$cluster_id)

Description

The canonical values of the vocabulary column of corpus$concepts. Readers write these names (for example "author-keywords" for author keywords and "wos-category" for the Web of Science categories), and the functions that select concepts by vocabulary, such as sm_to_bibliometrix() (DE) and sm_network_coword() (field = "keywords"), read them.

Each vocabulary belongs to a family:

• author_keyword: author-keywords, manual-tags;

• indexer_keyword: keywords-plus, index-keywords, automatic-tags, mesh;

• category: wos-category (WC, the Web of Science categories), wos-subject-category (SC, the research areas);

• concept: openalex, arxiv, biorxiv, medrxiv;

• custom: custom, for vocabularies of your own.

sm_corpus() and sm_corpus_from_tables() rename legacy and common spellings to these names, with a message: case, white space and the separators -, ⁠_⁠, . and / do not matter, "keywords-author", "keyword" and "keywords" become "author-keywords", "medline" becomes "mesh", and the Web of Science field tags "DE", "ID", "WC" and "SC" name their vocabularies. Other values are kept and named in the message. The functions that select concepts by vocabulary read these spellings too, so saved corpora and tables assigned to corpus$concepts work as well. The column stays character.

Usage

sm_concept_vocabularies(describe = FALSE)

Arguments

Value

A character vector (grouped by family) or a tibble of level/family/description.

See Also

sm_to_bibliometrix(), sm_network_coword(), sm_read_wos()

Other corpus: as_sm_corpus(), is_sm_corpus(), sm_bind_corpora(), sm_build_corpus(), sm_corpus(), sm_corpus_from_tables(), sm_dedupe(), sm_link_references(), sm_materialise(), sm_oa_statuses(), sm_save_corpus(), sm_validate(), validate_sm_corpus()

Examples

sm_concept_vocabularies()
sm_concept_vocabularies(describe = TRUE)

corpus <- sm_example_corpus(n_works = 5, with_embeddings = FALSE)
table(corpus$concepts$vocabulary)

Description

Build a typed, tibble-of-tibbles corpus container for bibliometric analysis. The corpus is the central data structure in scimapR, holding works, authors, authorships, institutions, sources, references, concepts, embeddings, provenance, screening decisions, and metadata.

Usage

sm_corpus(
  works,
  authors = NULL,
  authorships = NULL,
  institutions = NULL,
  sources = NULL,
  references = NULL,
  concepts = NULL,
  embeddings = NULL,
  provenance = NULL,
  screening = NULL,
  metadata = list(),
  citations_by_year = NULL
)

## S3 method for class 'sm_corpus'
x[i, ...]

## S3 method for class 'sm_corpus'
length(x)

## S3 method for class 'sm_corpus'
dim(x)

## S3 method for class 'sm_corpus'
as_tibble(x, ...)

## S3 method for class 'sm_corpus'
as.data.frame(x, ...)

## S3 method for class 'sm_corpus'
print(x, ...)

## S3 method for class 'sm_corpus'
format(x, ...)

## S3 method for class 'sm_corpus'
summary(object, ...)

## S3 method for class 'sm_corpus'
str(object, ...)

Arguments

Details

IDs are unique within a corpus; sm_bind_corpora() gives the second corpus new IDs where they collide.

A corpus is a list of tables that share identifiers: work_id links every table to works, author_id links authorships to authors, institution_id links them to institutions and source_id links works to sources. The tables below distinguish the standard schema columns (as in the empty tables) from optional columns that particular readers and functions write or read. A schema value may be NA. sm_corpus() adds missing schema columns to works (mostly NA); it takes other supplied tables as given. validate_sm_corpus() checks the required list elements, that works is a tibble with work_id, that metadata is a list, and the yearly citation schema below. It does not require or check every standard column of the other tables. Use sm_validate() for further data checks.

works: one row per work. Standard schema:

Optional: quartile, jif_quartile or sjr_quartile (journal quartiles such as "Q1", for sm_its(outcome = "share_q1")); numeric impact columns cnci, fwci, fnci, rcr, cnci_value, ncs, mncs and field_citation_ratio (searched in that order by sm_its() and sm_metric_summary()); field (for sm_metric_fnci(classification = "manual")); citation_mature and cnci_provisional (sm_citation_maturity()); cluster_id (the clustering functions); cluster_label and cluster_label_basis, a cluster's label and how it was made (sm_cluster_label()). The OpenAlex readers and sm_enrich_openalex_impact() write these impact columns:

The openalexR engine of sm_fetch_openalex() has no percentile, so its citation_percentile, is_top_10pct and is_top_1pct are NA. citations_as_of is the day of the OpenAlex record's updated_date, or of the fetch when the record has none; sm_read_openalex_json() leaves it NA for records without updated_date. sm_citation_window() reads it to tell complete citation windows from partial ones. A corpus restored with sm_corpus_from_tables() can hold it as a day count (days since 1970-01-01, as SQLite keeps dates) or as ISO 8601 text; sm_citation_window() and sm_enrich_openalex_impact() read both as the date they stand for.

authors: one row per author. Standard schema:

display_name_alternatives holds, for example, the names as a Web of Science file writes them, and sm_audit_gender(method = "manual") reads the gender columns. Optional: none so far.

authorships: one row per work, author and affiliation, so an author with several addresses in a work has several rows, which share position and is_corresponding. Standard schema:

is_corresponding is TRUE for the authors the source names as corresponding, FALSE for the other authors of such a work and NA when the source does not say who the corresponding author is. sm_read_wos() and as_sm_corpus() read it from the RP field, sm_read_openalex_json() and sm_fetch_openalex() from OpenAlex (a work whose authors are all marked false takes its corresponding authors from corresponding_author_ids, and counts as unknown when that lists none of them), sm_read_dimensions() from its Corresponding Authors column and sm_fetch_biorxiv() from the bioRxiv API; the other readers and fetchers write NA. Up to scimapR 0.4.0 they flagged the first author instead. A name that fits several authors of a work equally (namesakes with the same surname and initials) and that the source does not tell apart leaves them NA. Functions that select corresponding authors take TRUE only, and the leadership outcome of sm_its() leaves works whose corresponding author is unknown out of its shares.

Optional: institution_match, match_method, match_signal and match_evidence (sm_affiliation_match()); institution_name (sm_attribute_institution()); email (exported as EM by sm_to_bibliometrix()).

institutions: one row per institution. Standard schema:

Optional: none so far.

sources: one row per journal or other source. Standard schema:

Optional: abbreviation and iso_abbreviation (character; the Web of Science abbreviations J9 and JI, written by sm_read_wos() and the bibliometrix conversion and exported by sm_to_bibliometrix()).

references: one row per cited reference. Standard schema:

Reference linking: readers, fetchers, sm_bind_corpora(), sm_dedupe() and sm_link_references() set cited_work_id from cited_doi or an OpenAlex id in cited_raw when the cited work is in works.

Optional: none so far.

concepts: one row per keyword, category or concept of a work. Standard schema:

For MeSH headings score is 1 for a major topic and 0 otherwise. Optional: none so far.

embeddings: NULL or a numeric matrix with one row per embedded work, named by its work_id, and one column per dimension. Works may have no row: works without text to embed (sm_embed_works()), works missing from a cache (sm_embed_load()) or works for which a service has no vector (sm_enrich_specter()). The clustering functions give them no cluster (NA), and sm_network_semantic() and sm_plot_landscape() leave them out. metadata$embedding records the model (see below).

provenance: one row per work and source. Standard schema:

source_id_external is, for example, the Web of Science accession number (UT); prompt_hash is NA for records that no prompt produced. Optional:

kind is "api_query" for the searches of sm_fetch_openalex(), sm_fetch_pubmed(), sm_fetch_crossref() and sm_corpus_for_question(), which also write filter and n_max; "file" for the file readers, whose query holds the file relative to the working directory when it lies below it, else its name; and "enrichment" for the enrichers, such as sm_enrich_openalex_impact(), whose rows have the source "openalex_impact". sm_certificate() records the kind of each step, inferred from the source, query and engine for rows without kind, and sm_rebuild_from_cert() replays only searches, never a file read or an enrichment.

screening: one row per screening decision. Standard schema:

Optional: model records the chat-provider model for the decision. The legacy decision "unclear" is read as "uncertain".

citations_by_year: optional (NULL when the corpus has no yearly citation counts); one row per work and citation year, as the OpenAlex readers, sm_enrich_openalex_impact() and sm_corpus_from_tables() write it:

work_id and year may not be missing, and each pair appears once. OpenAlex gives the counts of the ten most recent years and leaves out the years without citations, so a missing year counts 0 for a work with yearly counts (see works$citations_as_of and sm_citation_window()). Subsetting keeps the rows of the kept works, sm_bind_corpora() binds them and sm_hash_corpus() includes them when there are any.

metrics: an optional legacy table read by the impact resolver when no impact column in works has values. It is not created by sm_corpus() or included in the standard empty schema:

The first named impact column, in the same order as for works and then value, is used when it contains a nonmissing value. Prefer impact columns in works for new corpora.

metadata: a list. Standard entries unless marked optional:

n_identified includes records removed by question-search deduplication or its result limit.

last_refresh advances only when sm_refresh() refreshed at least one work and no request failed. A recovered partial fetch has optional fetch_incomplete (list) with source, n_fetched, n_expected and reason; see the fetchers' Failed requests sections.

question_id is set by sm_corpus_for_question(). Optional entries locked_at (POSIXct), lock_reason (character) and lock_hash (character) record when, why and which content sm_lock() locked. Subsets remain locked but discard lock_hash. Other optional entries include those added by readers, such as source_file and reader; hash_version (integer), the algorithm of corpus_hash (sm_hash_corpus()), which sm_snapshot() writes (a snapshot without it used version 1); and local_inputs, a tibble of the local files the corpus was built from, with their SHA-256 hashes, which sm_register_input() and sm_corpus_from_tables() (inputs) write, sm_certificate() records and sm_verify_certificate() checks (see sm_register_input() for its columns). local_inputs is NULL until inputs are registered or supplied. build_date and last_refresh may also be numbers of seconds since 1970-01-01 UTC, as corpora restored from tables store them, ISO 8601 text or dates; sm_certificate() and the citation-maturity cut-offs read all of these. print() also reads these forms for last_refresh and shows ⁠YYYY-MM-DD HH:MM UTC⁠. NULL and a single missing value print as "never"; unsupported values and vectors use format().

Optional: embedding, the record of the model behind embeddings, a list that sm_embed_works(), sm_enrich_specter(), sm_fetch_semantic_scholar(), sm_embed_load() (from the cache of sm_embed_save()) and sm_bind_corpora() write and sm_certificate() reports:

model_id is the Hugging Face id of the model (for "specter2" that of the adapter, whose base model is base_model_id) or the Semantic Scholar field of the vectors ("embedding.specter_v2" or "embedding.specter_v1"), whose revision is the model version that Semantic Scholar reports. backend is "sentence-transformers", "adapters", "semantic-scholar" or "mixed"; a mix lists its models in models. prefix is "query: " for "multilingual-e5". A record whose n_dimensions differs from the width of embeddings, or whose n_works is smaller than its number of rows, no longer describes the matrix and counts as no record. Embeddings without a record, such as those made by earlier versions of scimapR, have a model that is not recorded: sm_certificate() reports it as "unknown", and the functions that combine embeddings combine them with recorded ones only when asked to (mix_models).

Value

An sm_corpus S3 object.

• [: An sm_corpus with the selected works and only the authors, institutions and sources they refer to. Counts and the corpus hash describe the subset, including when all works are selected.

• length(): Number of works (integer).

• dim(): Integer vector of length 2 (rows, columns of works table).

• as_tibble(): The works tibble.

• as.data.frame(): The works table as a data frame.

• print(): x invisibly.

• format(): A single character string.

• summary(): A named list of summary statistics.

• str(): object invisibly.

See Also

Other corpus: as_sm_corpus(), is_sm_corpus(), sm_bind_corpora(), sm_build_corpus(), sm_concept_vocabularies(), sm_corpus_from_tables(), sm_dedupe(), sm_link_references(), sm_materialise(), sm_oa_statuses(), sm_save_corpus(), sm_validate(), validate_sm_corpus()

Examples

corpus <- sm_corpus(
  works = tibble::tibble(
    work_id = "W000000001",
    doi = "10.1234/example",
    title = "Example Work",
    abstract = "An example abstract.",
    year = 2024L,
    type = "journal-article",
    source_id = NA_character_,
    cited_by_count = 0L,
    oa_status = "closed",
    language = "en",
    pmid = NA_character_,
    arxiv_id = NA_character_,
    openalex_id = NA_character_,
    is_retracted = FALSE,
    retraction_date = NA_real_,
    last_refreshed = Sys.time()
  )
)
print(corpus)

Description

Takes a structured sm_question() object, translates its fields into database-specific query strings, fetches results from the requested bibliographic sources, deduplicates on DOI, and returns an sm_corpus with metadata$question_id linked to the question.

This function requires the appropriate API-client packages to be installed for each source (openalexR for OpenAlex, rcrossref for Crossref, rentrez for PubMed).

Usage

sm_corpus_for_question(
  question,
  sources = c("pubmed", "openalex", "crossref"),
  n_max = 1000L,
  verbose = TRUE,
  call = rlang::caller_env()
)

Arguments

Value

An sm_corpus with metadata$question_id set to the question's content-hash ID. Its provenance records each search (query, n_max and, as engine, the client package that ran it), so that sm_rebuild_from_cert() can run it again.

See Also

Other question: is_sm_question(), sm_question(), sm_screen_against_question(), sm_screen_regex(), sm_screen_summary()

Examples

q <- sm_question(
  text = "Does immunotherapy improve survival in melanoma?",
  framework = "PICO",
  population = "melanoma",
  intervention = "immunotherapy",
  outcome = "survival"
)
# Would query live APIs:
# corpus <- sm_corpus_for_question(q)

Description

A documented, validating constructor that builds an sm_corpus from a named list of data frames – making non-OpenAlex/WoS tabular sources first-class and side-stepping format-specific parsers. Required columns are validated against the sm_corpus schema, column types are coerced (with informative messages), absent optional tables are filled with correctly-typed 0-row tibbles, and the result is validated before being returned. Legacy and common spellings of concept vocabularies (such as "keywords-author" or "Author Keywords") are renamed to the names of sm_concept_vocabularies(), with a message. Values that name no known vocabulary are kept as written and listed in the same message, each time a corpus is built from them; name a vocabulary of your own "custom".

This is the recommended ingestion path for arbitrary tabular sources.

Usage

sm_corpus_from_tables(
  tables,
  schema = NULL,
  .coerce = TRUE,
  metadata = list(),
  inputs = NULL,
  call = rlang::caller_env()
)

Arguments

Value

A validated sm_corpus object.

See Also

sm_corpus(), as_sm_corpus()

Other reporting: sm_figure_manifest()

Other corpus: as_sm_corpus(), is_sm_corpus(), sm_bind_corpora(), sm_build_corpus(), sm_concept_vocabularies(), sm_corpus(), sm_dedupe(), sm_link_references(), sm_materialise(), sm_oa_statuses(), sm_save_corpus(), sm_validate(), validate_sm_corpus()

Examples

works <- data.frame(
  work_id = c("W1", "W2"),
  title = c("First", "Second"),
  year = c("2020", "2021"),   # character -> coerced to integer
  doi = c("10.1/a", "10.1/b")
)
authorships <- data.frame(
  work_id = c("W1", "W1", "W2"),
  author_id = c("A1", "A2", "A1"),
  position = c(1, 2, 1)
)
corpus <- sm_corpus_from_tables(list(works = works,
                                     authorships = authorships))
corpus

# record the file the tables were read from
snapshot <- tempfile(fileext = ".csv")
utils::write.csv(works, snapshot, row.names = FALSE)
corpus <- sm_corpus_from_tables(
  list(works = utils::read.csv(snapshot)),
  inputs = c("source table" = snapshot)
)
corpus$metadata$local_inputs[, c("basename", "role", "sha256")]

Description

Attributes publication output (and citation impact) to entities using either full or fractional counting. Reviewers routinely ask for fractional counts so that a single multi-author / multi-institution paper is not counted in full for every contributor.

Usage

sm_count(
  corpus,
  method = c("full", "fractional"),
  level = c("institution", "author", "source"),
  call = rlang::caller_env()
)

Arguments

Details

The fractional rule is the standard equal-share rule: a work with $k$ distinct entities contributes $1/k$ of credit (and $1/k$ of its citations) to each. For level = "source" each work has a single source, so fractional and full counts coincide.

For level = "institution" each authorship row is labelled on its own, by the first of these authorships columns that has a value in that row: institution_match (from sm_affiliation_match()), institution_name, institution_id (named by corpus$institutions$display_name when that table lists the id) and raw_affiliation. A row labelled through institution_id has that id as entity_id; every other row has its label as both entity_id and entity_name. Rows labelled by their raw affiliation string are counted un-disambiguated, with a warning, because sm_count() applies no affiliation dictionary itself: run sm_affiliation_match() with a dictionary of your institutions first to count them. The order is the one sm_coverage_audit() uses for its affiliation levels, which however takes one column for all rows and falls back to country codes instead of raw strings. Up to scimapR 0.4.0, sm_count() used one column for every row (the first of institution_id, institution_match and raw_affiliation with any value), so rows without a value in it were left out, and it matched raw strings against the shipped sm_affiliation_dict before counting them.

sm_attribute_institution() keeps institution_match, so its roll-up of the matches to a controlled vocabulary (ROR ids and names in institution_id and institution_name) is counted only for rows without a match; up to scimapR 0.4.0 the counts followed institution_id. To count by the vocabulary, put its names first:

a <- corpus$authorships
corpus$authorships$institution_match <-
  dplyr::coalesce(a$institution_name, a$institution_match)

Value

A tibble, one row per entity, sorted by credit descending:

entity_id

Entity identifier.

entity_name

Human-readable name (falls back to entity_id).

n_works

Number of distinct works the entity appears on (this is the full count and is identical for both methods).

credit

Output credit: n_works under "full"; the sum of per-work fractional shares under "fractional".

weighted_citations

Citation impact attributed to the entity: summed cited_by_count under "full"; summed fractional-share weighted cited_by_count under "fractional".

Type-stable: an empty/inapplicable corpus returns a 0-row tibble with these columns.

See Also

sm_metric_summary(), sm_affiliation_match()

Other counting: sm_citation_maturity(), sm_citation_window(), sm_metric_summary()

Examples

corpus <- sm_example_corpus(n_works = 30, seed = 1)
sm_count(corpus, method = "fractional", level = "author")

# Institutions: match the affiliation strings with a dictionary of your
# institutions first, then count
a <- corpus$authorships
corpus$authorships$raw_affiliation <- ifelse(
  a$position == 1L,
  "Unit North, Northtown",
  "Department of Surgery, Unit South"
)
dict <- list("Unit North" = "unit north", "Unit South" = "unit south")
matched <- sm_affiliation_match(corpus, patterns = dict)
sm_count(matched, method = "fractional", level = "institution")

Description

Computes recall, precision, and F1 of an sm_corpus against an external ground-truth reference (a manual tracker, an ORCID works set, an institutional-repository export, or another sm_corpus). This is the core primitive for a coverage / completeness audit: "how much of what we should have did we actually capture, and how much of what we captured is real?"

Matching is by normalised DOI with a fuzzy-title fallback (see sm_reconcile() for the shared matching engine). Full match provenance is retained so every decision can be inspected.

Usage

sm_coverage_audit(
  corpus,
  reference,
  by = NULL,
  match = c("doi_then_title", "doi", "title"),
  threshold = 0.9,
  index_table = NULL,
  call = rlang::caller_env()
)

## S3 method for class 'sm_coverage'
print(x, ...)

## S3 method for class 'sm_coverage'
summary(object, ...)

## S3 method for class 'sm_coverage'
autoplot(object, dim = NULL, ...)

Arguments

Details

Named dimensions. "year" uses the publication years. "source" uses the source names of an sm_corpus (sources$display_name, or works$source_id when the corpus has no source table) and, in a data frame, the first of the columns source, source_title, journal, so and source_id, in this order (names matched case-insensitively); a source_id column holds ids, whose levels do not join the names of the corpus side. "affiliation" uses, in a data frame, the first of the columns affiliation, affiliations, institution, country and country_code, in this order. A data-frame cell holds one value, so an item reported by two units takes two rows; a list column gives each of its values. Missing and blank values have no level. A named dimension that neither side can resolve gives a warning and no breakdown rows. Only "affiliation" joins a tracker's units with the institutions of the corpus: by = "institution" names the column itself, which the corpus side has only if works has a column of that name.

Affiliation levels of a corpus. For an sm_corpus (the corpus under audit, or a reference given as one), "affiliation" labels the authorships from the first of these authorships columns that has a value in at least one row: institution_match (from sm_affiliation_match()), institution_name (from sm_attribute_institution()), institution_id (named by corpus$institutions$display_name when that table lists the id, else the id itself) and, as a last resort, country_code. One column serves every row, so the levels of a side are all of one kind, and an authorship without a value in that column has no level. A message names the column used on each side, and a warning says when a side falls back to country codes, whose levels do not join institution names. Because sm_attribute_institution() runs sm_affiliation_match() first, institution_name is used only when institution_match is absent or empty; the Details of sm_attribute_institution() show how to break the coverage down by its vocabulary instead. sm_count(level = "institution") reads the same columns in the same order, but row by row, and counts the raw affiliation string where none has a value instead of falling back to countries; up to scimapR 0.4.0 it used, for every row, the first of institution_id, institution_match and the raw strings (matched with sm_affiliation_dict) that had a value in any row.

Full counting. A record belongs to every level of its values, so a work with authors at two institutions counts in both slices, and the per-level counts of a dimension can add up to more than n_corpus or n_reference. The overall recall and precision count each record once.

Value

An sm_coverage S3 object (a list) with components:

recall

Reference records found in the corpus / total reference records. Each reference row is looked up on its own, so rows that report the same item (one paper reported by two units) are all found.

precision

Matched corpus records / total corpus records. Each corpus record pairs with at most one reference row.

f1

Harmonic mean of recall and precision.

n_corpus, n_reference

Integer record counts.

n_matched

Reference records found in the corpus (the numerator of recall).

n_corpus_matched

Corpus records paired with a reference record (the numerator of precision); equal to n_matched unless reference rows share a corpus record.

n_corpus_only, n_reference_only

Unmatched counts.

matches

Tibble with one row per corpus record: corpus_id, reference_id, match_type ("doi"/"title"/"none"), match_score. When index_table is supplied, also issn, indexable (logical), and indexed_title.

reference_matches

Tibble with one row per reference record: reference_id, corpus_id (the corpus record it was found in), match_type and match_score; the recall-side counterpart of matches.

corpus_only

Tibble of corpus records absent from the reference.

reference_only

Tibble of reference records absent from the corpus.

breakdowns

A single flat tibble (dimension, level, n_reference, n_matched, recall) across all by dimensions, from the reference side. Use sm_coverage_breakdowns() to access/filter it, also with the corpus-side counts and precision.

breakdowns_nested

The legacy named list of per-dimension tibbles (slice, n_reference, n_matched, recall). Retained for one release; prefer breakdowns.

indexability

When index_table is supplied, a summary tibble of indexable vs non-indexable record counts; otherwise NULL.

print returns x invisibly.

summary returns a one-row tibble of headline metrics: recall, precision, f1, n_corpus, n_reference, n_matched, n_corpus_only, n_reference_only and n_corpus_matched, so that n_matched + n_reference_only gives n_reference and n_corpus_matched + n_corpus_only gives n_corpus.

autoplot returns a ggplot object.

See Also

sm_reconcile(), sm_journal_in_index(), sm_coverage_breakdowns(), sm_affiliation_match()

Other coverage: sm_coverage_breakdowns(), sm_journal_in_index(), sm_match_types(), sm_reconcile()

Examples

corpus <- sm_example_corpus(n_works = 30, seed = 1)
# Pretend the reference is the corpus minus a few works, plus an extra one
ref <- corpus$works[1:25, c("work_id", "doi", "title", "year")]
cov <- sm_coverage_audit(corpus, ref, by = "year")
cov

# Any column of the reference or of the works is a dimension too
ref$unit <- rep(c("Unit A", "Unit B"), length.out = nrow(ref))
cov <- sm_coverage_audit(corpus, ref, by = c("unit", "type"), match = "doi")
sm_coverage_breakdowns(cov)

Description

Accessor returning the per-dimension coverage breakdowns of an sm_coverage_audit() result as a single flat (long) tibble, optionally filtered to one dimension.

Usage

sm_coverage_breakdowns(x, dimension = NULL, tidy = TRUE)

Arguments

Details

This accessor honours the package's accessor return-type stability contract (see scimapR-stability): with tidy = TRUE the column set is guaranteed and will only change via a lifecycle deprecation.

The recall columns (n_reference, n_matched, recall) come from the reference side and the precision columns (n_corpus, precision) from the corpus side, joined on ⁠(dimension, level)⁠. A level found on the reference side only has n_corpus, precision and f1 NA; a level found on the corpus side only has n_reference, n_matched, recall and f1 NA. Under full counting (see the Details of sm_coverage_audit()) the per-level counts of a dimension can add up to more than the record totals. tidy = FALSE, print() and autoplot() show the same reference-side counts.

Value

A tibble (see tidy). Type-stable: a 0-row tibble with the documented columns when there are no breakdowns. Slices present on only one side (corpus vs reference) carry NA in the other side's counts/metric.

See Also

sm_coverage_audit(), scimapR-stability

Other coverage: sm_coverage_audit(), sm_journal_in_index(), sm_match_types(), sm_reconcile()

Examples

corpus <- sm_example_corpus(n_works = 30, seed = 1)
ref <- corpus$works[1:25, c("work_id", "doi", "title", "year")]
cov <- sm_coverage_audit(corpus, ref, by = "year")
sm_coverage_breakdowns(cov, dimension = "year")

Description

Remove duplicate works, preferring the record with more complete data. Ties keep the first record. Kept records stay in their input order, with only their child rows and referenced lookup records.

Usage

sm_dedupe(corpus, by = "doi", verbose = TRUE)

Arguments

Value

An sm_corpus object.

See Also

Other corpus: as_sm_corpus(), is_sm_corpus(), sm_bind_corpora(), sm_build_corpus(), sm_concept_vocabularies(), sm_corpus(), sm_corpus_from_tables(), sm_link_references(), sm_materialise(), sm_oa_statuses(), sm_save_corpus(), sm_validate(), validate_sm_corpus()

Description

Fits a difference-in-differences (DiD) model comparing the yearly outcome series of a treated institution set against a control set, before and after intervention_year. Group membership is resolved via the institution-attribution columns from sm_affiliation_match() / sm_attribute_institution().

Usage

sm_did(
  corpus,
  treated,
  control,
  intervention_year,
  outcome = c("count", "share_q1", "cnci", "leadership"),
  family = NULL,
  lag = NULL,
  institution_aware = FALSE,
  lead_role = c("corresponding", "first", "last", "first_or_last", "any_lead"),
  shared = c("keep", "exclude", "lead"),
  min_cell = NULL,
  call = rlang::caller_env()
)

## S3 method for class 'sm_did'
print(x, ...)

## S3 method for class 'sm_did'
summary(object, ...)

## S3 method for class 'sm_did'
autoplot(object, ...)

Arguments

Details

Each group's yearly series is built as in sm_its(). For "count" a group's years without works count 0, over every year from the first to the last publication year of the corpus. For the other outcomes a year in which a group has works but no usable value is dropped from that group's series, with a warning, and listed in dropped_years. For "cnci" the citation-immature years are then excluded from both series, by the same rule as in sm_its(), and listed in provisional_years; up to scimapR 0.4.0 sm_did() kept them. Each group needs 4 or more years, with at least one before intervention_year and one from it on; otherwise sm_did() aborts. The confidence interval uses the reference distribution of the p-value (t with the residual degrees of freedom for the default Gaussian family); no correction for autocorrelation is applied.

A work belongs to a group when one of its authorships carries one of the group's labels, so a work co-authored by both groups belongs to both. Such shared works pull the two series together and the estimate toward 0. sm_did() warns when there are shared works and shared is not set, and records them in shared_works. For "leadership" the denominators leave out the works whose lead is unknown, as in sm_its().

Value

An sm_did S3 object with components: model, did_estimate (a one-row tibble: estimate, std.error, conf.low, conf.high, p.value), series (group-by-year tibble: year, group, value, n_works), dropped_years (tibble: group, year; the group-years left out for lack of a usable value), provisional_years (integer; the citation-immature years excluded for "cnci"), shared_works (tibble: work_id, series; the works of both groups and where they went: "both", "treated", "control" or "none"), shared (the policy), leadership (the leadership definition, as in sm_its(), with institutions the labels of each group when institution_aware; NULL for the other outcomes), impact_source (where the impact values of "cnci" come from, as in sm_its(); NA for the other outcomes), and metadata (outcome, intervention_year, family, lag).

print returns x invisibly.

summary returns the DiD estimate tibble.

autoplot returns a ggplot (parallel trends + divergence).

See Also

sm_its(), sm_synth()

Other causal: sm_its(), sm_synth()

Examples

corpus <- sm_example_corpus(n_works = 120, seed = 1)
# Tag two institution groups for illustration: each work belongs to one
a <- corpus$authorships
group <- rep(c("Inst A", "Inst B"), length.out = nrow(corpus$works))
corpus$authorships$institution_name <-
  group[match(a$work_id, corpus$works$work_id)]
did <- sm_did(corpus, treated = "Inst A", control = "Inst B",
              intervention_year = 2020, outcome = "count")
did

Description

Produces a detailed comparison between two sm_corpus objects, reporting works added, removed, and changed; author differences; reference differences; and screening changes. This is useful for auditing how a corpus evolved between snapshots or after a refresh.

This function compares strictly by internal work_id. For content-based reconciliation across corpora that do not share identifiers (matching by normalised DOI with a fuzzy title fallback), use sm_reconcile().

Usage

sm_diff_corpora(corpus1, corpus2, call = rlang::caller_env())

## S3 method for class 'sm_corpus_diff'
print(x, ...)

Arguments

Value

An sm_corpus_diff S3 object (a list) with components:

added

Tibble of works in corpus2 but not corpus1.

removed

Tibble of works in corpus1 but not corpus2.

changed

Tibble of works present in both but with differing fields.

summary

A one-row summary tibble with counts.

hash1

Hash of corpus1.

hash2

Hash of corpus2.

See Also

sm_reconcile()

Other reproducibility: sm_certificate(), sm_cite_corpus(), sm_hash_corpus(), sm_provenance(), sm_register_input(), sm_snapshot()

Examples

c1 <- sm_example_corpus(seed = 1L)
c2 <- sm_example_corpus(seed = 2L)
d <- sm_diff_corpora(c1, c2)
print(d)

Description

Loads a previously saved embeddings matrix from an RDS file and attaches it to the corpus. Only embeddings for works present in the corpus are loaded; mismatches are reported.

The loaded embeddings replace any the corpus had, and so does the record of their model (corpus$metadata$embedding, see sm_corpus()): the cache's record when it describes the cached matrix, else none, so the model of embeddings from an older cache or a bare matrix counts as not recorded ("unknown" in sm_certificate()).

Usage

sm_embed_load(corpus, path, call = rlang::caller_env())

Arguments

Value

The input corpus with corpus$embeddings and corpus$metadata$embedding updated. Works without a cached row get no row, so the clustering functions give them no cluster (NA).

See Also

Other embedding: sm_embed_save(), sm_embed_works()

Examples

## Not run: 
corpus <- sm_example_corpus(with_embeddings = FALSE)
corpus <- sm_embed_load(corpus, "my_embeddings.rds")
dim(corpus$embeddings)

## End(Not run)

Description

Saves the embeddings matrix from an sm_corpus to an RDS file on disk. This allows caching expensive embedding computations for later reuse with sm_embed_load(). The cache keeps the record of the model behind the embeddings (corpus$metadata$embedding, see sm_corpus()) when the corpus has one.

Usage

sm_embed_save(corpus, path, call = rlang::caller_env())

Arguments

Value

The input corpus (invisibly).

See Also

Other embedding: sm_embed_load(), sm_embed_works()

Examples

## Not run: 
corpus <- sm_example_corpus(with_embeddings = TRUE)
sm_embed_save(corpus, "my_embeddings.rds")

## End(Not run)

Description

Generates dense vector embeddings for works in an sm_corpus using pretrained transformer models in Python. The default model, specter2, loads the SPECTER2 adapter with the Python package adapters; the other models load with sentence-transformers. Requires reticulate and a Python installation with the packages the chosen model needs.

Usage

sm_embed_works(
  corpus,
  model = c("specter2", "scincl", "scibert", "minilm-l6", "mpnet", "multilingual-mpnet",
    "multilingual-e5"),
  text = c("title_abstract", "title", "abstract"),
  batch_size = 32L,
  cache_dir = tools::R_user_dir("scimapR", "cache"),
  python = NULL,
  install_deps = FALSE,
  verbose = TRUE,
  call = rlang::caller_env()
)

Arguments

Details

Models and their Hugging Face models:

• "specter2": allenai/specter2, an adapter on allenai/specter2_base;

• "scincl": malteos/scincl;

• "scibert": allenai/scibert_scivocab_uncased;

• "minilm-l6": sentence-transformers/all-MiniLM-L6-v2;

• "mpnet": sentence-transformers/all-mpnet-base-v2;

• "multilingual-mpnet": sentence-transformers/paraphrase-multilingual-mpnet-base-v2, trained on more than 50 languages;

• "multilingual-e5": intfloat/multilingual-e5-base, trained on about 100 languages.

The first five are trained on English text. "specter2" loads with the Python package adapters, the others with sentence-transformers.

"specter2" follows its model card: the base model with the proximity adapter, title and abstract joined by the separator token ⁠[SEP]⁠ without spaces, at most 512 tokens, and the first token of the last layer as the embedding. It needs the Python packages torch, transformers and adapters; the other models need sentence-transformers. A missing package stops the function with an error naming it, and so does a model that does not load; it never falls back to another model.

The other models join title and abstract with a space. For "multilingual-e5" every text starts with "query: ", the prefix its model card gives for clustering and for using the embeddings as features; its embeddings have unit length. "multilingual-mpnet" reads at most 128 tokens, so it sees the title and the start of a long abstract only. "scibert" has no sentence-transformers configuration, so sentence-transformers averages its token vectors, with a warning.

When more than 10% of the works with a known language (works$language: names such as "German", ISO 639-1 codes such as "de" or ISO 639-2 codes such as "ger") are not in English and an English-only model is chosen, a message (class scimapR_embed_multilingual) suggests a multilingual model; it changes nothing else.

Works without text to embed (with "title_abstract" neither a title nor an abstract, with "title" or "abstract" no such field; blank text counts as none) would all get the same vector and become each other's nearest neighbours. They are left out, with one warning (class scimapR_embed_no_text) giving their number: they get no row in corpus$embeddings, so the clustering functions give them no cluster (NA), and sm_network_semantic() and sm_plot_landscape() leave them out, as they do for works missing from a cache loaded with sm_embed_load().

corpus$metadata$embedding records what produced the embeddings: the model, its Hugging Face id and, when the local model snapshot in cache_dir shows it, its revision, the text used, the input prefix and the versions of the Python packages (see sm_corpus()). sm_certificate() reports it in embedding_info.

Python and its packages are soft dependencies (reticulate is in Suggests). If they are unavailable, the function aborts with an informative message.

Value

The input corpus with corpus$embeddings replaced by a numeric matrix with one row per embedded work and one column per dimension, and with corpus$metadata$embedding recording the model. Row names are the work_ids of the embedded works: every work that has text to embed (see Details).

See Also

Other embedding: sm_embed_load(), sm_embed_save()

Examples

## Not run: 
corpus <- sm_example_corpus(with_embeddings = FALSE)
corpus <- sm_embed_works(corpus, model = "minilm-l6")
dim(corpus$embeddings)
corpus$metadata$embedding$model_id

# a corpus with works in several languages
corpus <- sm_embed_works(corpus, model = "multilingual-mpnet")

## End(Not run)

Description

Look up DOIs in the corpus via the Altmetric API and add attention scores, Mendeley reader counts, and social media metrics to the works table.

This enricher is idempotent: re-running it updates existing Altmetric data rather than duplicating it.

Usage

sm_enrich_altmetric(
  corpus,
  api_key = Sys.getenv("ALTMETRIC_API_KEY"),
  verbose = TRUE,
  call = rlang::caller_env()
)

Arguments

Value

An sm_corpus object with added altmetric_score, mendeley_readers, twitter_count, and news_count columns in the works table, plus new provenance rows.

See Also

Other enrichers: sm_enrich_concepts(), sm_enrich_openalex_impact(), sm_enrich_opencitations(), sm_enrich_orcid(), sm_enrich_retraction(), sm_enrich_ror(), sm_enrich_specter(), sm_enrich_unpaywall()

Examples

## Not run: 
corpus <- sm_example_corpus()
corpus <- sm_enrich_altmetric(corpus)

## End(Not run)

Description

Look up works in the corpus via OpenAlex or PubMed to retrieve associated concepts, topics, or MeSH terms and add them to the concepts table.