misskappa estimates weighted agreement coefficients (Conger, Fleiss, Brennan–Prediger) and coefficient alpha when some ratings are missing, and attaches an asymptotic covariance matrix to every estimate so you get standard errors, confidence intervals, and equality tests for free. Each estimator is chosen with a single estimator= argument. You never have to remember a combination of method and type flags.
A first agreement coefficient
dat.gwet2014 is a small categorical rating matrix (20 subjects, five raters, four categories) with some ratings missing (NA). The inverse-probability-weighted (IPW) estimator is consistent under missing-completely-at-random missingness:
fit <- kappa(dat.gwet2014, estimator = "ipw", weight = "linear")
#> Warning: rater pair(s) rater4-rater5 co-observed by only one subject; the
#> corresponding pairwise covariance is degenerate and the standard error
#> unreliable.
fit
#> misskappa: estimator=ipw, weight=linear
#> estimate se lower upper
#> Conger 0.5879 0.0772 0.4365 0.7393
#> Fleiss 0.5858 0.0774 0.4341 0.7375
#> Brennan-Prediger 0.6397 0.0767 0.4894 0.7900The print method shows each coefficient with its standard error and a 95% confidence interval. The object carries everything you need:
Confidence intervals
confint() gives natural-scale Wald intervals, or a variance-stabilizing Fisher z interval that always stays inside (-1, 1), useful when a coefficient is close to the boundary:
Choosing an estimator
The right estimator= depends on the data and the missingness assumption:
| Input | estimator |
Assumption |
|---|---|---|
| Categorical ratings | "ipw" |
MCAR, rater-varying observation rates |
| Categorical ratings | "cat_fiml" |
Ignorable (MCAR or MAR), saturated multinomial |
| Scored ratings (quadratic) | "pairwise" |
MCAR, distribution-free |
| Scored ratings (quadratic) | "nt_fiml" |
Ignorable, normal-theory |
The scored estimators treat the categories as numbers and use the quadratic loss. Pass the scores directly, or integer codes together with values.
kappa(dat.zapf2016, estimator = "pairwise")
#> misskappa: estimator=pairwise, weight=quadratic
#> estimate se lower upper
#> Conger 0.8985 0.0278 0.8439 0.953
#> Fleiss 0.8984 0.0279 0.8437 0.953Counts-format data
When raters are interchangeable and you only have per-subject category counts, use kappa_counts(). dat.fleiss1971 is the classic psychiatric-diagnosis table (30 subjects, six raters, five categories):
kappa_counts(dat.fleiss1971, estimator = "fleiss_cuzick")
#> misskappa: estimator=fleiss_cuzick, weight=nominal
#> estimate se lower upper
#> Fleiss 0.4302 0.0533 0.3258 0.5347
#> Brennan-Prediger 0.4444 0.0542 0.3382 0.5507Coefficient alpha
alpha() estimates Cronbach’s coefficient alpha for a subjects-by-items score battery, again with missing entries allowed. psych::bfi is a 25-item Big Five questionnaire answered by 2800 people, many of whom skipped at least one item. Treating the five-item Neuroticism scale as the battery, the normal-theory FIML estimator handles the missing responses under ignorable missingness:
data(bfi, package = "psych")
neuroticism <- bfi[, paste0("N", 1:5)]
sum(!complete.cases(neuroticism)) # respondents missing at least one item
#> [1] 106
alpha(neuroticism, estimator = "nt_fiml")
#> misskappa: estimator=nt_fiml, weight=score
#> estimate se lower upper
#> alpha 0.8138 0.006 0.802 0.8256Beyond pairwise: g-wise and vector-valued agreement
The same kappa() reaches two generalizations. Set g > 2 for g-wise agreement, where the disagreement kernel compares g raters at once (g = 2 is ordinary pairwise kappa):
kappa(dat.gwet2014, estimator = "ipw", weight = "nominal", g = 3)
#> Warning: rater 3-tuple(s) rater1-rater4-rater5, rater2-rater4-rater5,
#> rater3-rater4-rater5 co-observed by only one subject; the corresponding
#> observed-disagreement term is degenerate and the standard error unreliable.
#> misskappa: estimator=ipw, weight=nominal, g=3
#> estimate se lower upper
#> Conger 0.4369 0.0850 0.2703 0.6034
#> Fleiss 0.4308 0.0861 0.2620 0.5997And when each rating is a vector, pass a subjects-by-raters-by-features array. dat.vanbelle2019 holds 28 observers classifying crackles at six chest sites for 20 patients. A component loss (weight) measures vector disagreement:
kappa(dat.vanbelle2019[, 1:4, ], estimator = "pairwise")
#> misskappa: estimator=pairwise, weight=hamming
#> estimate se lower upper
#> Conger 0.5192 0.0794 0.3635 0.6748
#> Fleiss 0.5150 0.0813 0.3557 0.6743The g-wise agreement article develops the multirater kernels in full. Vector-valued agreement is frontier / experimental ongoing research — see the vector-valued agreement article.
Testing whether coefficients are equal
A common question is whether a coefficient is the same across two groups, two estimators, two timepoints, or several rater pairs. kappa_test() and alpha_test() answer it and return a standard htest. Use paired = TRUE (the default) when the fits share the same subjects (the dependence is taken from the per-subject influence functions) and paired = FALSE for independent samples.
kappa_test(AB = kappa(dat.zapf2016[, 1:2], estimator = "pairwise"),
CD = kappa(dat.zapf2016[, 3:4], estimator = "pairwise"),
coef = "Conger")
#>
#> Paired (dependent) test of equal Conger across 2 fits
#>
#> data: AB, CD
#> X-squared = 2.2432, df = 1, p-value = 0.1342
#> sample estimates:
#> AB CD
#> 0.9309091 0.8466753See the equality tests article for the one-sample, independent, paired, and G-way homogeneity cases worked out on real data, and the mathematical guides for the loss-matrix formulation and the missing-data estimators.