Ordinary kappa compares two scalar ratings at a time. This article covers the two ways agreement() goes beyond that: g-wise kernels that compare g raters at once, and vector-valued ratings.
g-wise agreement
Conger’s and Fleiss’ kappa are already multirater coefficients: with more than two raters they average a pairwise disagreement kernel over every rater pair and correct it against a pairwise chance baseline. So “many raters” by itself is not a new construction. It is what the pairwise estimators in the getting-started vignette already do.
The genuine generalization is to let the disagreement kernel look at more than two raters at once. This is g-wise agreement: the kernel is a symmetric function of g ratings, and g (its arity) is the new knob. The default g = 2 recovers ordinary pairwise kappa. g > 2 measures how often a whole group of g raters fails to coincide, which a sequence of pairwise comparisons cannot express. The kernels come from the Fréchet / Hubert multirater family [Hubert 1977; Moss 2024].
A first g-wise coefficient
# Pairwise (g = 2) versus triple-wise (g = 3) nominal agreement on the
# Gwet (2014) categorical data (five raters, some ratings missing).
agreement(dat.gwet2014, estimator = "ipw", weight = "nominal")
#> Warning: rater pair(s) rater4-rater5 co-observed by only one subject; the
#> corresponding pairwise covariance is degenerate and the standard error
#> unreliable.
#> misskappa: estimator = ipw, weight = nominal, n = 20
#> estimate se lower upper
#> Conger 0.4233 0.1119 0.2040 0.6426
#> Fleiss 0.4183 0.1131 0.1966 0.6399
#> Brennan-Prediger 0.4421 0.1212 0.2045 0.6797
agreement(dat.gwet2014, estimator = "ipw", weight = "nominal", g = 3)
#> Warning: rater 3-tuple(s) rater1-rater4-rater5, rater2-rater4-rater5,
#> rater3-rater4-rater5 co-observed by only one subject; the corresponding
#> observed-disagreement term is degenerate and the standard error unreliable.
#> misskappa: estimator = ipw, weight = nominal, g = 3, n = 20
#> estimate se lower upper
#> Conger 0.4369 0.0850 0.2703 0.6034
#> Fleiss 0.4308 0.0861 0.2620 0.5997The g = 3 coefficient is not a reweighting of the pairwise one. It asks a strictly harder question (do three raters coincide, not just two), so it is a different estimand with its own standard error.
Mathematical description
Fix a symmetric g-rater disagreement kernel \(d(x_{1},\ldots,x_{g}) \ge 0\) that vanishes when all g arguments are equal. For a subject with \(R\) raters, the observed disagreement averages this kernel over the \(\binom{R}{g}\) within-subject rater combinations,
\[ D_{d} \;=\; \binom{R}{g}^{-1} \sum_{j_{1}<\cdots<j_{g}} d\!\left(X_{j_{1}},\ldots,X_{j_{g}}\right). \]
Chance correction replaces the within-subject raters by raters drawn from independent subjects, which removes any real agreement and leaves only what the rater margins produce by coincidence. The Conger-type baseline keeps the raters distinct,
\[ C_{d} \;=\; \binom{R}{g}^{-1} \sum_{j_{1}<\cdots<j_{g}} \mathbb{E}\, d\!\left(X_{j_{1}}^{(1)},\ldots,X_{j_{g}}^{(g)}\right), \]
with each \(X^{(t)}\) from an independent copy of the subject. The Fleiss-type baseline instead pools the rater margins, averaging over all \(R^{g}\) ordered rater tuples. The two coefficients are then
\[ \kappa_{d} \;=\; 1 - \frac{D_{d}}{C_{d}}, \qquad \pi_{d} \;=\; 1 - \frac{D_{d}}{F_{d}}, \]
the Conger (\(\kappa_d\)) and Fleiss (\(\pi_d\)) g-wise coefficients reported above. For \(g = 2\) these are exactly Conger’s and Fleiss’ kappa.
The kernels themselves are Fréchet variances [Moss 2024]. Given a scalar loss \(l(\mu, x)\), the Fréchet variance of a configuration is the minimal average loss to a common point,
\[ V(l)\!\left[x_{1},\ldots,x_{g}\right] \;=\; \min_{\mu}\; \frac{1}{g}\sum_{t=1}^{g} l\!\left(\mu, x_{t}\right), \]
and taking \(d = V(l)\) supplies a symmetric multirater kernel for any \(l\). With \(g = 2\) this is half the pairwise loss, so the construction is a genuine extension of the loss-matrix picture in the coefficients and estimators article. Nominal and absolute losses give the "nominal" and "linear" kernels. The all-raters-equal kernel of Hubert (1977) is exposed as "hubert".
Estimators and weights
Two estimators carry over from the pairwise case. "ipw" reweights for missing entries under MCAR and reduces to the complete-data estimator when no rating is missing. "cat_fiml" is the saturated-multinomial FIML for categorical ratings under ignorable missingness. For g > 2 the supported weights are "nominal", "linear" (with "ipw" only), and "hubert" (the all-raters-equal kernel, defined only for g > 2).
Quadratic weighting is g-invariant
Quadratic (squared-error) disagreement is special: it decomposes into pairwise terms, so the combinatorial constant cancels in the kappa ratio and the g-wise coefficient equals the pairwise one for every g. agreement() exploits this: with weight = "quadratic" the g argument is ignored and the cheap closed form is used instead of the \(g^{\text{th}}\)-order enumeration.
g2 <- coef(agreement(dat.gwet2014, estimator = "pairwise", weight = "quadratic"))
#> Warning: rater pair(s) rater4-rater5 co-observed by only one subject; the
#> corresponding pairwise covariance is degenerate and the standard error
#> unreliable.
g4 <- coef(agreement(dat.gwet2014, estimator = "pairwise", weight = "quadratic", g = 4))
#> Warning: rater pair(s) rater4-rater5 co-observed by only one subject; the
#> corresponding pairwise covariance is degenerate and the standard error
#> unreliable.
rbind(g2 = g2, g4 = g4)
#> Conger Fleiss
#> g2 0.7517242 0.7503656
#> g4 0.7517242 0.7503656Because the quadratic coefficient is a smooth functional of the rater means and covariance, it also admits the normal-theory FIML estimator (estimator = "nt_fiml") under ignorable missingness, exactly as in the scalar pairwise case. There is no separate g-wise machinery to invoke.
Vector-valued agreement
In preparation
The inference here is ordinary. The same
pairwiseandipwestimators as scalaragreement(), applied one component at a time, so a missing component just drops from the rater pairs it touches. For the quadratic weight the coefficient is Lin’s concordance correlation coefficient, and fitting it to incomplete data is the variance-components approach long used for that coefficient under unbalanced designs (King & Chinchilli 2001; Carrasco & Jover 2003).What is still in preparation is the construction around that inference: the component-separable weighted family below, the full feature-coupling matrix \(W\), and the supporting manuscript. The interface may change.
Sometimes a single rating is itself a vector. In the CRACKLES auscultation study shipped as dat.vanbelle2019, each of 28 observers classifies the presence of crackles (0/1) at six chest sites per patient, so one observer’s verdict on one patient is a six-component vector. The data are a subjects-by-raters-by-features array, and an array keeps its labels on every axis:
Pass the array and agreement() takes the vector-valued path automatically.
Mathematical description
Write a rater’s vector rating on feature \(\ell\) as \(x_\ell\), with non-negative feature weights \(v_\ell\). The disagreement between two rater vectors \(x\) and \(y\) is a component-separable loss
\[ D(x, y) \;=\; h\!\left( \frac{\sum_{\ell=1}^{p} v_\ell\, d_\ell(x_\ell, y_\ell)} {\sum_{\ell=1}^{p} v_\ell} \right), \]
a weighted average of per-component losses \(d_\ell\) passed through a scalar transform \(h\). The weight argument selects the component loss: "nominal" is Hamming (count of disagreeing sites, \(d_\ell = \mathbf 1[x_\ell \neq y_\ell]\), \(h(t)=t\)), "linear" is absolute difference, and "quadratic" is squared difference. Passing loss = "rms" keeps the squared component loss but sets \(h(t)=\sqrt t\). Per-component weights enter through feature_weights, e.g. to down-weight some sites.
The Conger and Fleiss coefficients are then the usual \(\kappa = 1 - D_o/D_e\), with observed disagreement, distinct-rater-pair chance, and pooled-margin chance all built from these component losses. The separability is deliberate. Because disagreement is a sum over features, a missing component simply drops that feature from the affected rater pair, and the estimator stays a ratio of feature-level moments. The closest published precedent is the single fixed-loss multivariate agreement coefficient of Janson & Olsson (2001).
The expert panel
agreement(dat.vanbelle2019[, 1:4, ], estimator = "pairwise")
#> misskappa: estimator = pairwise, weight = nominal, n = 20
#> estimate se lower upper
#> Conger 0.5192 0.0794 0.3635 0.6748
#> Fleiss 0.5150 0.0813 0.3557 0.6743The Conger coefficient by observer group reproduces the kind of breakdown Vanbelle (2019) reports, with experts and pulmonologists agreeing most and students least:
groups <- c("EXP", "NOR", "RUS", "WAL", "NLD", "PUL", "STU")
conger <- sapply(groups, function(grp) {
cols <- paste0(grp, 1:4)
coef(agreement(dat.vanbelle2019[, cols, ], estimator = "pairwise"))["Conger"]
})
round(sort(conger, decreasing = TRUE), 3)
#> NOR.Conger EXP.Conger WAL.Conger NLD.Conger PUL.Conger STU.Conger RUS.Conger
#> 0.535 0.519 0.506 0.478 0.384 0.354 0.187For missing components, use estimator = "ipw", which reweights by the per-rater-feature observation rates. Two knobs ride along through ...: feature_weights (non-negative per-site weights) and loss = "rms" (the root-mean-square component loss). On the complete CRACKLES data IPW agrees with the pairwise estimator:
The quadratic case
With the squared component loss the construction generalizes further. Allow a full symmetric positive-semidefinite feature-weight matrix \(W\), so that features may be coupled rather than weighted one at a time,
\[ D(x, y) \;=\; (x - y)^{\top} W (x - y), \]
which reduces to the separable squared loss when \(W\) is diagonal. As in the scalar quadratic kappa, the coefficient is then a smooth function of the stacked rater mean \(\mu\) and covariance \(\Sigma\): with \(B_W = \sum_{j,k}\operatorname{tr}(W\Sigma_{jk})\), \(T_W = \sum_{j}\operatorname{tr}(W\Sigma_{jj})\), and \(G_W = \sum_{j}(\mu_j-\bar\mu)^{\top} W (\mu_j-\bar\mu)\),
\[ \kappa_{C}^{W} = \frac{B_W - T_W}{(R-1)\,T_W + R\,G_W}, \qquad \kappa_{F}^{W} = \frac{B_W - T_W - G_W}{(R-1)\,(T_W + G_W)}. \]
Because the estimand depends on the data only through \((\mu, \Sigma)\), it admits a FIML estimator. Fit the saturated normal mean and covariance by EM and use every partially observed vector, exactly as the scalar nt_fiml quadratic kappa does.
Passing a feature metric W to agreement() selects this squared loss. "pairwise" is the moment estimator, "nt_fiml" the saturated normal FIML, and "nt_fiml_cs" the FIML under the block compound-symmetry covariance \(\Sigma = J_R \otimes \Psi + I_R \otimes E\), the vector analogue of the scalar variance-component model. With the identity metric, the moment and compound-symmetry estimates agree on the complete CRACKLES data:
sites <- dat.vanbelle2019[, 1:4, ]
agreement(sites, estimator = "pairwise", W = diag(6))
#> misskappa: estimator = pairwise, weight = quadratic, n = 20
#> estimate se lower upper
#> Conger 0.5192 0.0794 0.3635 0.6748
#> Fleiss 0.5150 0.0813 0.3557 0.6743
agreement(sites, estimator = "nt_fiml_cs", W = diag(6))
#> misskappa: estimator = nt_fiml_cs, weight = quadratic, n = 20
#> estimate se lower upper
#> Conger 0.5192 0.0794 0.3635 0.6748
#> Fleiss 0.5150 0.0813 0.3557 0.6743The saturated "nt_fiml" inverts per-pattern covariance blocks and fails with a numerical error when the fitted covariance is singular, which binary data with few observers can trigger. The compound-symmetry model has far fewer parameters and is the practical FIML choice there.
Brennan-Prediger needs a declared rating scale. For vector ratings, pass ranges, a features-by-2 matrix of (lo, hi) bounds, to the FIML estimators. It fixes a continuous uniform chance baseline per feature, which suits continuous scores rather than binary ones like these.
References
Carrasco, J. L., & Jover, L. (2003). Estimating the generalized concordance correlation coefficient through variance components. Biometrics, 59(4), 849–858. https://doi.org/10.1111/j.0006-341X.2003.00099.x
Conger, A. J. (1980). Integration and generalization of kappas for multiple raters. Psychological Bulletin, 88(2), 322–328.
Hubert, L. (1977). Kappa revisited. Psychological Bulletin, 84(2), 289–297.
Janson, H., & Olsson, U. (2001). A measure of agreement for interval or nominal multivariate observations. Educational and Psychological Measurement, 61(2), 277–289.
King, T. S., & Chinchilli, V. M. (2001). A generalized concordance correlation coefficient for continuous and categorical data. Statistics in Medicine, 20(14), 2131–2147. https://doi.org/10.1002/sim.845
Lin, L. I.-K. (1989). A concordance correlation coefficient to evaluate reproducibility. Biometrics, 45(1), 255–268. https://doi.org/10.2307/2532051
Moss, J. (2024). Measures of agreement with multiple raters: Fréchet variances and inference. Psychometrika. https://doi.org/10.1007/s11336-023-09945-2
Vanbelle, S. (2019). Asymptotic variability of (multilevel) multirater kappa coefficients. Statistical Methods in Medical Research, 28(10–11), 3012–3026. https://doi.org/10.1177/0962280218794733
