
Synthetic planned-incomplete ratings (compound-symmetry illustration)
Source:R/datasets.R
dat.tenhove2025.RdA small synthetic inter-rater dataset built to illustrate the
compound-symmetry (variance-component) estimators. It is not real data:
60 subjects are scored on an ordinal 1–5 scale by 6 raters drawn from a
fixed-rater variance-component model y_ij = mu_j + b_i + e_ij, in the style
of the planned-incomplete ICC designs of ten Hove, Jorgensen, and van der
Ark (2025). Each subject is rated by only three of the six raters, in three
overlapping panels (raters 1,2,3, then 3,4,5, then 5,6,1), so the rater
co-observation graph is connected but not complete. Some rater pairs
(for example raters 2 and 4) never rate the same subject. That is exactly the
regime where the saturated coefficients are not identified: agreement() with
estimator = "pairwise", "nt_fiml", or "cat_fiml" stops with an
identifiability error, while the compound-symmetry estimator
"nt_fiml_cs", which needs only a connected graph, still fits. The build
script data_raw/tenhove2025.R regenerates it from a fixed
seed.
Format
A 60 x 6 integer matrix. Columns rater1–rater6 hold ordinal
scores 1–5, with NA marking the (planned) unobserved ratings. Every
row has exactly three observed entries.
Source
Simulated. Inspired by the planned-incomplete ICC designs of ten Hove et al. (2025). No real measurements are used.
References
ten Hove, D., Jorgensen, T. D., & van der Ark, L. A. (2025). How to Estimate ICCs for Interrater Reliability from Planned Incomplete Data. Multivariate Behavioral Research, 60(5). doi:10.1080/00273171.2025.2507745
Examples
# The co-observation graph is connected but incomplete, so the saturated
# estimators are not identified ...
try(agreement(dat.tenhove2025, estimator = "nt_fiml"))
#> Error : rater pair(s) rater1-rater4, rater2-rater4, rater2-rater5, rater2-rater6, rater3-rater6, rater4-rater6 never jointly observed, so Conger's kappa (and Brennan-Prediger) is not identified from this missing-data pattern (the co-observation graph is incomplete). Drop an offending rater, or provide data in which every rater pair is co-observed.
# ... while the compound-symmetry estimator needs only connectivity.
agreement(dat.tenhove2025, estimator = "nt_fiml_cs", values = 1:5)
#> misskappa: estimator = nt_fiml_cs, weight = quadratic, n = 60
#> estimate se lower upper
#> Conger 0.6845 0.0539 0.5789 0.7902
#> Fleiss 0.6837 0.0542 0.5773 0.7900
#> Brennan-Prediger 0.8120 0.0264 0.7603 0.8637