Estimates Conger, Fleiss, and Brennan-Prediger agreement coefficients (with
two raters, Conger is Cohen's kappa) from a subjects-by-raters matrix in
which some ratings may be missing. Choose the estimator with estimator:
"ipw"(default, recommended): inverse-probability weighting. Consistent under MCAR, also with raters observed at different rates; works with anyweight."cat_fiml": categorical full-information maximum likelihood. Consistent under MAR; feasible for few raters and categories."pairwise": moment estimator for quadratic weights, MCAR."nt_fiml": robust normal-theory FIML for quadratic weights, suited to MAR with roughly normal scores. Falls back to"pairwise"when the fit fails; see the section on the fallback."nt_fiml_cs":"nt_fiml"under the compound-symmetry covariance \(\Sigma = \psi J + e I\); needs only a connected rater co-observation graph, so it handles planned-incomplete designs.
The last three treat ratings as numeric scores and use the quadratic weight.
Arguments
- x
A subjects-by-raters matrix or data frame of ratings, or a subjects-by-raters-by-features array of vector-valued ratings.
NAmarks missing entries.- estimator
"ipw"(default),"cat_fiml","pairwise","nt_fiml", or"nt_fiml_cs".- weight
"nominal"(default),"linear", or"quadratic". The scored estimators always use"quadratic"."hubert"is an all-raters-equal kernel forg > 2. For vector-valued ratings it selects the component loss (Hamming, absolute, or squared).- g
Number of raters the disagreement kernel compares at once.
2(default) is ordinary kappa. Quadratic agreement does not depend ong.- values
Optional numeric scores, one per category in scale order. Defaults to the categories themselves when numeric, else
1:C.- categories
Optional vector declaring the full rating scale in order, for example
1:5orc("low", "mid", "high").- em_options
Named list of EM settings for the likelihood estimators:
tolandmax_iter, and for"cat_fiml"alsoprune_tol,start_alpha,info_rcond, andflatten.info_rcondis the relative eigenvalue cutoff of the information matrix used for the standard errors.flatten > 0adds a small Dirichlet pseudo-count that makes the fitted table unique when the likelihood is flat; the estimate barely moves but standard errors become conservative. Settings an estimator does not use are an error.- fallback
For
estimator = "nt_fiml"only:"pairwise"(the default) returns the"pairwise"estimate when the fit fails or does not converge;"none"raises the error. See the section on the fallback.- ...
Arguments for vector-valued ratings:
W(a feature metric for the squared loss),ranges(a features-by-2 matrix of score bounds, for Brennan-Prediger),feature_weights, andloss = "rms". Forg > 2:max_chance_tuples(default5e6). Anything else is an error.
Value
A misskappa_estimate: the estimates, their covariance matrix,
the number of subjects n, the rating scale categories, and the
per-subject influence rows psi, with vcov = crossprod(psi) / n^2.
"cat_fiml" fits also carry two diagnostics per coefficient:
null_frac, the share of the coefficient's gradient that lies in
directions the likelihood does not identify, and se_cutoff_ratio, the
standard error recomputed with info_rcond lowered 100-fold divided by
the reported one (values well above 1 mean the standard error leans on
weakly identified directions of a sparse table).
Methods: print, coef, vcov, confint, as.data.frame, and
stats::influence.
Rating scale
Ratings may be numbers, strings, logicals, or factors. The scale defaults
to the shared factor levels, or else the sorted observed ratings. Text
ratings have no order of their own, so linear and quadratic weights and the
scored estimators need categories (or factors with shared levels) for
them. Pass
categories to declare the full scale including categories nobody used
(categories = 1:5 on a 1-5 scale where no one chose 5). The scale sets
Brennan-Prediger's chance baseline and the range that scales the linear and
quadratic weights. "nt_fiml" and "nt_fiml_cs" report Brennan-Prediger
only when categories, values, or factor levels declare the scale;
"pairwise" reports Conger and Fleiss only.
Missing data
Subjects with no ratings are dropped. Every rater must be observed, and
every rater pair must rate at least one subject together (every g-tuple for
g > 2), otherwise the coefficient is not identified and the call errors.
"nt_fiml_cs" needs only a connected co-observation graph.
Fallback for "nt_fiml"
The normal-theory likelihood breaks down when the fitted covariance of the
ratings is singular or nearly so, for example when two raters agree
perfectly on the subjects they share. Kappa itself is still well defined
there, but the maximum-likelihood fit and its sandwich standard error are
not. This is common in small samples with high agreement and in planned
designs where each rater pair shares few subjects. On complete data
"nt_fiml" equals "pairwise" and never fails.
By default (fallback = "pairwise"), a failed or non-converged "nt_fiml"
fit is replaced by the "pairwise" estimate of the same coefficient, with
a warning and the reason recorded in fit$fallback. "pairwise" is the
moment estimator that "nt_fiml" reduces to on complete data, so the
combination is consistent under MCAR everywhere and keeps "nt_fiml"'s
advantage under MAR whenever the fit succeeds. In the simulation study of
Moss (in preparation) the combination had smaller bias at n = 40 than
either estimator alone, because "nt_fiml" fails selectively on
high-agreement samples. Use fallback = "none" to get the error instead.
Generalizations
g > 2 uses g-wise kernels that compare g raters at once (with "ipw"
or "cat_fiml"). A subjects-by-raters-by-features array gives
vector-valued agreement; see the beyond-pairwise article.
References
Moss, J. (in preparation). Agreement coefficients with missing ratings.
Moss, J. (in preparation). Quadratically weighted agreement coefficients.
See also
agreement_counts() for counts-format data, agreement_test()
to compare coefficients, confint.misskappa_estimate().
Examples
# Five raters, three categories, some ratings missing.
agreement(dat.klein2018)
#> misskappa: estimator = ipw, weight = nominal, n = 10
#> estimate se lower upper
#> Conger 0.4301 0.1050 0.2244 0.6358
#> Fleiss 0.4054 0.1196 0.1710 0.6399
#> Brennan-Prediger 0.4204 0.1136 0.1978 0.6430
# Ordered categories with linear weights, on a declared 1-4 scale.
agreement(dat.klein2018, weight = "linear", categories = 1:4)
#> misskappa: estimator = ipw, weight = linear, n = 10
#> estimate se lower upper
#> Conger 0.3398 0.1389 0.0675 0.6121
#> Fleiss 0.3026 0.1624 -0.0157 0.6210
#> Brennan-Prediger 0.4764 0.1269 0.2278 0.7251
# Quadratic agreement on a 1-5 scale under MAR, normal theory.
agreement(dat.zapf2016, estimator = "nt_fiml", categories = 1:5)
#> misskappa: estimator = nt_fiml, weight = quadratic, n = 50
#> estimate se lower upper
#> Conger 0.8985 0.0278 0.8439 0.9530
#> Fleiss 0.8984 0.0279 0.8437 0.9530
#> Brennan-Prediger 0.8675 0.0313 0.8062 0.9288
# A planned-incomplete design: some rater pairs never meet, so only the
# compound-symmetry estimator is identified.
agreement(dat.tenhove2025, estimator = "nt_fiml_cs")
#> misskappa: estimator = nt_fiml_cs, weight = quadratic, n = 60
#> estimate se lower upper
#> Conger 0.6845 0.0539 0.5789 0.7902
#> Fleiss 0.6837 0.0542 0.5773 0.7900
