Analysing two Delphi rounds offline

This guide executes an offline analysis workflow with synthetic ratings. It needs neither Shiny nor PostgreSQL. It does not demonstrate recruitment, data collection, or production approval. The consensus thresholds are examples; a real study must justify and specify its rules in advance.

Validate the protocol and snapshot

library(delphyr)

config <- demo_protocol()
validate_protocol(config)$valid
#> [1] TRUE
protocol <- new_protocol(config)
protocol$analysis$consensus$group_policy
#> [1] "all_required_groups"

The default protocol requires adequate results in every designated panel group (all_required_groups). The compact snapshot example instead explicitly uses pooled: all ten ratings come from one group. That choice is frozen in its protocol; the analysis does not silently adjust the rule after seeing results.

first <- demo_snapshot()
first$protocol$analysis$consensus$group_policy
#> [1] "pooled"
first$items
#>   item_code item_version dimension_code  scale_code
#> 1      I001            1      relevance relevance_9

new_snapshot() accepts responses, assignments, an instrument, and a validated protocol. Its content hash binds those inputs to the round number. Construct a new snapshot to change inputs; the analysis rejects content that no longer matches its recorded hash.

Read results with their denominators

analysis <- analyse_round(first)
analysis$decisions
#>   item_code item_version dimension_code classification
#> 1      I001            1      relevance   consensus_in
analysis$results[, c("stratum", "n_assigned", "n_submitted", "n_valid",
                     "n_agree", "n_disagree", "p_agree", "classification")]
#>               stratum n_assigned n_submitted n_valid n_agree n_disagree p_agree
#> 1             overall         10          10      10       7          1     0.7
#> 2       professionals         10          10      10       7          1     0.7
#> 3 public_contributors          0           0       0       0          0      NA
#>      classification
#> 1      consensus_in
#> 2      consensus_in
#> 3 insufficient_data

The ten valid ratings contain seven values in 7–9, one in 1–3, and two in the middle range. Their unrounded proportions are 0.7 and 0.1, producing consensus_in under this example rule. The empty public_contributors group remains visible as insufficient_data. Pooled consensus does not demonstrate consensus within every group.

required_groups <- protocol$analysis$consensus
analyse_round(first, rules = required_groups)$decisions
#>   item_code item_version dimension_code    classification
#> 1      I001            1      relevance insufficient_data

The explicitly supplied alternative rule returns insufficient_data. Its own rule hash is recorded in the analysis; it does not alter the source snapshot. A prespecified primary analysis should use the frozen study protocol.

with_missing <- analyse_round(demo_snapshot(c(NA, 4, 6, 7, 7, 8, 8, 9, 9, 9)))
with_missing$denominators
#>   item_code item_version dimension_code             stratum n_assigned
#> 1      I001            1      relevance             overall         10
#> 2      I001            1      relevance       professionals         10
#> 3      I001            1      relevance public_contributors          0
#>   n_submitted n_valid n_agree n_disagree
#> 1          10       9       7          0
#> 2          10       9       7          0
#> 3           0       0       0          0
with_missing$missingness
#>   item_code item_version dimension_code             stratum n_not_answered
#> 1      I001            1      relevance             overall              1
#> 2      I001            1      relevance       professionals              1
#> 3      I001            1      relevance public_contributors              0
#>   n_unable n_abstained n_not_applicable
#> 1        0           0                0
#> 2        0           0                0
#> 3        0           0                0
with_missing$decisions
#>   item_code item_version dimension_code    classification
#> 1      I001            1      relevance insufficient_data

In this helper, NA becomes an explicit not_answered status. Nine valid ratings fall below the example minimum of ten. Assigned people, submissions, and special response categories are counted separately; missingness does not become a zero rating. Median and quartiles use valid ratings, with type 7 quantiles.

Construct and compare a second round

The demo helper creates round 1. For the second round, construct a new snapshot with round_number = 2L. Participant identifiers and item meaning stay the same in this example.

second_demo <- demo_snapshot(c(2, 4, 7, 7, 7, 8, 8, 9, 9, 9))
response_columns <- c("panelist_id", "item_code", "item_version", "dimension_code",
                      "answer_status", "value_integer", "value_text")
assignment_columns <- c("panelist_id", "group_code", "submitted")
second <- new_snapshot(
  responses = second_demo$data[, response_columns],
  assignments = unique(second_demo$data[, assignment_columns]),
  items = second_demo$items,
  protocol = second_demo$protocol,
  round_number = 2L
)

analyse_round(second)$decisions
#>   item_code item_version dimension_code classification
#> 1      I001            1      relevance   consensus_in
comparison <- compare_rounds(first, second)
comparison[, c("status", "n_paired", "n_lost", "n_new",
                "median_absolute_change", "proportion_unchanged")]
#>        status n_paired n_lost n_new median_absolute_change proportion_unchanged
#> 1 descriptive       10      0     0                      0                  0.8

Ten people are paired and eight ratings remain unchanged. Descriptive change does not automatically establish stability or end a study; stable disagreement is possible. Unpaired round means must not be interpreted as individual change.

When an item’s meaning changes, exclude it from direct comparison. An explicit mapping can document that review decision even when version numbers match:

mapping <- data.frame(
  item_code = "I001", dimension_code = "relevance",
  previous_version = 1L, current_version = 1L,
  comparable = FALSE, reason = "Comparison excluded after methodological review"
)
compare_rounds(first, second, mapping)[, c("status", "reason", "n_paired")]
#>           status                                          reason n_paired
#> 1 not_comparable Comparison excluded after methodological review        0

New item versions are not comparable by default without an appropriate explicit mapping. A mapping records a methodological decision; it does not replace review.

Prepare a feedback draft

feedback <- prepare_feedback(analysis)
validate_feedback(feedback)
#> [1] TRUE
feedback$results[, c("stratum", "n_valid", "classification", "suppressed")]
#>   stratum n_valid classification suppressed
#> 1 overall      10   consensus_in      FALSE

By default, the draft contains only the overall result. Small groups must not be reconstructed by subtracting subgroup results from the total. When group statistics are enabled, the current conservative policy suppresses every result for an item if any displayed stratum falls below the minimum:

group_feedback <- prepare_feedback(analysis, policy = list(
  group_statistics = TRUE,
  minimum_display_cell_n = 5L,
  complementary_suppression = TRUE
))
group_feedback$results[, c("stratum", "n_valid", "classification", "suppressed")]
#>               stratum n_valid classification suppressed
#> 1             overall      NA           <NA>       TRUE
#> 2       professionals      NA           <NA>       TRUE
#> 3 public_contributors      NA           <NA>       TRUE

This cell rule is not a general anonymity guarantee. Qualitative summaries need review and a source reference. A draft is not yet an approved, immutable participant feedback release.

Check reproducibility and provenance

identical(analysis, analyse_round(first))
#> [1] TRUE
analysis$provenance[c("software_version", "quantile_type", "snapshot_hash",
                      "rules_hash", "result_hash")]
#> $software_version
#> [1] "0.0.2"
#> 
#> $quantile_type
#> [1] 7
#> 
#> $snapshot_hash
#> [1] "v2:96114993c64ab26110254d2c20043194f00ce5eb6ba08d7e0f43a786b06ed0f5"
#> 
#> $rules_hash
#> [1] "v2:bcf2ac52b830b1419e27cc0128837e8ce9a31e6b660bc75759b3925916f9a94d"
#> 
#> $result_hash
#> [1] "v2:5c84470e5108cb28c3794fde2072fb4bf618369cbf9bb5bcf937ddfeb6dbd288"

The same frozen inputs and rules produce the same result in the same software environment. Hashes detect changes; they do not replace access control or methodological review. Sharing results also requires the relevant data approval and export profile.

Reproduce a private research export

The database export uses the same analyse_round() logic. Its manifest includes snapshot.json, analysis tables, a dictionary, frozen instrument texts, and the HTML report. After extracting an authorized export, reproduce_export() checks its file hashes and recomputes the analysis without the application or Quarto:

# A previously authorized local export, not an example downloaded from a public URL.
reproduced <- reproduce_export("/path/to/extracted-export")
reproduced$provenance$result_hash

Report services require an authorized database connection and are separate from the executable offline example above. Reports distinguish frozen research data from editorial records observed at export time. Missing author information stays explicitly undocumented. Qualitative originals and individual free-text answers are excluded from the numeric profile. A research export is not a public-release approval. Stored protocol and instrument wording retain their authored language; the documentation does not automatically translate scientific content.

sessionInfo()
#> R version 4.6.2 (2026-10-29)
#> Platform: x86_64-pc-linux-gnu
#> Running under: Ubuntu 26.04.1 LTS
#> 
#> Matrix products: default
#> BLAS:   /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3 
#> LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.32.so;  LAPACK version 3.12.0
#> 
#> locale:
#>  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
#>  [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
#>  [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
#>  [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
#>  [9] LC_ADDRESS=C               LC_TELEPHONE=C            
#> [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       
#> 
#> time zone: Etc/UTC
#> tzcode source: system (glibc)
#> 
#> attached base packages:
#> [1] stats     graphics  grDevices utils     datasets  methods   base     
#> 
#> other attached packages:
#> [1] delphyr_0.0.2  rmarkdown_2.32
#> 
#> loaded via a namespace (and not attached):
#>  [1] digest_0.6.39    R6_2.6.1         fastmap_1.2.0    xfun_0.62       
#>  [5] maketools_1.3.2  cachem_1.1.0     knitr_1.52       htmltools_0.5.9 
#>  [9] buildtools_1.0.0 lifecycle_1.0.5  cli_3.6.6        sass_0.4.10     
#> [13] jquerylib_0.1.4  compiler_4.6.2   sys_3.4.3        tools_4.6.2     
#> [17] evaluate_1.0.5   bslib_0.12.0     yaml_2.3.12      otel_0.2.0      
#> [21] jsonlite_2.0.0   rlang_1.3.0      uuid_1.2-2