Synthetic planned-incomplete ratings (compound-symmetry illustration)

A small *synthetic* inter-rater dataset built to illustrate the compound-symmetry (variance-component) estimators. It is **not real data**: 60 subjects are scored on an ordinal `1`–`5` scale by 6 raters drawn from a fixed-rater variance-component model `y_ij = mu_j + b_i + e_ij`, in the style of the *planned-incomplete* ICC designs of ten Hove, Jorgensen, and van der Ark (2025). Each subject is rated by only three of the six raters, in three overlapping panels (raters `1,2,3`, then `3,4,5`, then `5,6,1`), so the rater co-observation graph is *connected but not complete*. Some rater pairs (for example raters 2 and 4) never rate the same subject. That is exactly the regime where the saturated coefficients are not identified: `kappa()` with `estimator = "pairwise"`, `"nt_fiml"`, or `"cat_fiml"` stops with an identifiability error, while the compound-symmetry estimators `"nt_fiml_cs"` and `"pairwise_cs"`, which need only a connected graph, still fit. The build script `data_raw/tenhove2025.R` regenerates it from a fixed seed.

Usage

dat.tenhove2025

Format

A 60 x 6 integer matrix. Columns `rater1`–`rater6` hold ordinal scores `1`–`5`, with `NA` marking the (planned) unobserved ratings. Every row has exactly three observed entries.

Source

Simulated. Inspired by the planned-incomplete ICC designs of ten Hove et al. (2025). No real measurements are used.

References

ten Hove, D., Jorgensen, T. D., & van der Ark, L. A. (2025). How to Estimate ICCs for Interrater Reliability from Planned Incomplete Data. Multivariate Behavioral Research, 60(5). doi:10.1080/00273171.2025.2507745

Examples

# The co-observation graph is connected but incomplete, so the saturated
# estimators are not identified ...
try(kappa(dat.tenhove2025, estimator = "nt_fiml"))
#> Error : rater pair(s) rater1-rater4, rater2-rater4, rater2-rater5, rater2-rater6, rater3-rater6, rater4-rater6 never jointly observed, so Conger's kappa (and Brennan-Prediger) is not identified from this missing-data pattern (the co-observation graph is incomplete). Drop an offending rater, or provide data in which every rater pair is co-observed.

# ... while the compound-symmetry estimators need only connectivity.
kappa(dat.tenhove2025, estimator = "nt_fiml_cs")
#> misskappa: estimator=nt_fiml_cs, weight=quadratic
#>                  estimate     se  lower  upper
#> Conger             0.6845 0.0539 0.5789 0.7902
#> Fleiss             0.6837 0.0542 0.5773 0.7900
#> Brennan-Prediger   0.7180 0.0395 0.6405 0.7955
kappa(dat.tenhove2025, estimator = "pairwise_cs")
#> misskappa: estimator=pairwise_cs, weight=quadratic
#>                  estimate     se  lower  upper
#> Conger             0.6920 0.0541 0.5859 0.7981
#> Fleiss             0.6920 0.0542 0.5858 0.7983
#> Brennan-Prediger   0.7301 0.0390 0.6537 0.8065