Skip to contents

A small synthetic inter-rater dataset built to illustrate the compound-symmetry (variance-component) estimators. It is not real data: 60 subjects are scored on an ordinal 1–5 scale by 6 raters drawn from a fixed-rater variance-component model y_ij = mu_j + b_i + e_ij, in the style of the planned-incomplete ICC designs of ten Hove, Jorgensen, and van der Ark (2025). Each subject is rated by only three of the six raters, in three overlapping panels (raters 1,2,3, then 3,4,5, then 5,6,1), so the rater co-observation graph is connected but not complete. Some rater pairs (for example raters 2 and 4) never rate the same subject. That is exactly the regime where the saturated coefficients are not identified: agreement() with estimator = "pairwise", "nt_fiml", or "cat_fiml" stops with an identifiability error, while the compound-symmetry estimator "nt_fiml_cs", which needs only a connected graph, still fits. The build script data_raw/tenhove2025.R regenerates it from a fixed seed.

Usage

dat.tenhove2025

Format

A 60 x 6 integer matrix. Columns rater1–rater6 hold ordinal scores 1–5, with NA marking the (planned) unobserved ratings. Every row has exactly three observed entries.

Source

Simulated. Inspired by the planned-incomplete ICC designs of ten Hove et al. (2025). No real measurements are used.

References

ten Hove, D., Jorgensen, T. D., & van der Ark, L. A. (2025). How to Estimate ICCs for Interrater Reliability from Planned Incomplete Data. Multivariate Behavioral Research, 60(5). doi:10.1080/00273171.2025.2507745

Examples

# The co-observation graph is connected but incomplete, so the saturated
# estimators are not identified ...
try(agreement(dat.tenhove2025, estimator = "nt_fiml"))
#> Error : rater pair(s) rater1-rater4, rater2-rater4, rater2-rater5, rater2-rater6, rater3-rater6, rater4-rater6 never jointly observed, so Conger's kappa (and Brennan-Prediger) is not identified from this missing-data pattern (the co-observation graph is incomplete). Drop an offending rater, or provide data in which every rater pair is co-observed.

# ... while the compound-symmetry estimator needs only connectivity.
agreement(dat.tenhove2025, estimator = "nt_fiml_cs", values = 1:5)
#> misskappa: estimator = nt_fiml_cs, weight = quadratic, n = 60
#>                  estimate     se  lower  upper
#> Conger             0.6845 0.0539 0.5789 0.7902
#> Fleiss             0.6837 0.0542 0.5773 0.7900
#> Brennan-Prediger   0.8120 0.0264 0.7603 0.8637