# Each subject has exactly three of the six ratings.
table(rowSums(is.finite(dat.tenhove2025)))
#>
#> 3
#> 60
# Off-diagonal zeros are rater pairs that never rate the same subject.
co <- crossprod(is.finite(dat.tenhove2025))
diag(co) <- 0
co
#> rater1 rater2 rater3 rater4 rater5 rater6
#> rater1 0 20 20 0 20 20
#> rater2 20 0 20 0 0 0
#> rater3 20 20 0 20 20 0
#> rater4 0 0 20 0 20 0
#> rater5 20 0 20 20 0 20
#> rater6 20 0 0 0 20 0Coefficients
Agreement coefficients in misskappa are written in terms of pairwise disagreement. For two raters with categorical ratings in \(\{1,\ldots,C\}\), let \(L\) be a loss matrix with \(\ell_{cc}=0\) and \(0 \le \ell_{cd} \le 1\). The generic coefficient is
\[ \kappa = 1 - \frac{D_o}{D_e}, \]
where \(D_o\) is observed disagreement and \(D_e\) is the disagreement expected under a chance model. Different named coefficients mostly differ in how they estimate these two quantities when there are more than two raters.
For a subject \(i\) and raters \(r,s\), complete-data pairwise disagreement is
\[ d_{irs} = \ell_{X_{ir}, X_{is}}. \]
Observed disagreement \(D_o\) averages \(d_{irs}\) over subjects and rater pairs. The coefficients differ in the chance model behind \(D_e\): Conger pairs each rater’s own category distribution, Fleiss pools the raters’ distributions, and Brennan-Prediger draws uniformly from the rating scale, so it needs that scale to be known (declare it with categories when some categories went unused).
The weight argument picks the loss: "nominal" (\(\ell_{cd}=1\) for \(c \neq d\)), or "linear" and "quadratic" for ordered categories, scaled by the range of the category scores \(v\). The quadratic loss is
\[ \ell_{cd} = \frac{(v_c - v_d)^2}{(\max_j v_j - \min_j v_j)^2}, \]
and lets the coefficient be expressed through the rater means and covariance matrix, which is what the scored estimators "pairwise", "nt_fiml", and "nt_fiml_cs" exploit: they estimate those moments under missing data and map them to the coefficients.
Estimators under missing data
The estimators split into moment estimators, consistent when ratings are missing completely at random (MCAR), and likelihood estimators, consistent when missingness may depend on the observed ratings (MAR). NA marks a missing rating.
The simplest moment estimator averages the loss over the observed rater pairs:
\[ \widehat D_o = \frac{\sum_i \sum_{r<s} R_{ir}R_{is} \ell_{X_{ir},X_{is}}} {\sum_i \sum_{r<s} R_{ir}R_{is}}, \]
where \(R_{ir}=1\) when the rating is observed. Pooling pairs this way weights each rater pair by how often it is co-observed. That is harmless when raters share an observation rate, but with rater-varying rates and non-exchangeable raters it targets a different average than the fixed-rater coefficient.
The IPW estimator ("ipw") corrects this by reweighting observed pairs by estimated observation probabilities. In schematic form,
\[ \widehat D_o^{\mathrm{IPW}} = \frac{1}{n}\sum_i \sum_{r<s} \frac{R_{ir}R_{is}}{\widehat\pi_r\widehat\pi_s} \ell_{X_{ir},X_{is}}, \]
with corresponding weighted marginal estimates for chance disagreement. This is the default and recommended estimator for categorical ratings. The scored "pairwise" estimator takes a different route to the same goal: it estimates each rater mean and each pairwise covariance from the subjects who observed it, and the quadratic coefficients are functions of those moments, so it is consistent under MCAR.
The categorical FIML estimator ("cat_fiml") fits a categorical full-data distribution under MAR by EM. For observed pattern \(o_i\), the likelihood contribution is
\[ L_i(\theta) = \sum_{x_{\mathrm{mis}}} p_\theta(x_{\mathrm{obs},i}, x_{\mathrm{mis}}), \]
and agreement coefficients are computed from the fitted full-data distribution. The implementation returns Wald covariance estimates through Louis’ observed information when available.
For scored (quadratically weighted) ratings the normal-theory FIML fits a Gaussian mean and covariance by EM and reads the coefficient off the fitted moments with a sandwich delta-method standard error. Two covariance models are available, selected by the estimator name: "nt_fiml" fits a saturated covariance, while "nt_fiml_cs" fits the compound-symmetry \(\Sigma = \psi J + e I\) (the fixed-rater variance-component model, with free rater means). Compound symmetry needs only a connected co-observation graph and is consistent and slightly more efficient when that structure holds, and is robust under MCAR even when the covariance is misspecified. The saturated model is the safer choice under MAR when the covariance may depart from compound symmetry.
Compound symmetry on a planned-incomplete design
The saturated coefficients ("pairwise", "nt_fiml", "cat_fiml") are functions of every pairwise second moment, so they are identified only when every rater pair is jointly observed by some subject, a complete co-observation graph. The compound-symmetry estimator needs only a connected graph, and that is the whole point of a planned-incomplete design: you can rate each subject with a rotating subset of raters and still recover the coefficient.
The synthetic dat.tenhove2025 data (60 subjects, 6 raters, ordinal 1–5) is such a design, in the style of the planned-incomplete ICC studies of ten Hove et al. (2025). Each subject is rated by three of the six raters in overlapping panels, so the graph is connected but some pairs (raters 2 and 4, for example) never co-occur:
Asking for a saturated coefficient on this pattern stops with an identifiability error before the estimator runs:
agreement(dat.tenhove2025, estimator = "nt_fiml")
#> Error:
#> ! rater pair(s) rater1-rater4, rater2-rater4, rater2-rater5, rater2-rater6, rater3-rater6, rater4-rater6 never jointly observed, so Conger's kappa (and Brennan-Prediger) is not identified from this missing-data pattern (the co-observation graph is incomplete). Drop an offending rater, or provide data in which every rater pair is co-observed.The compound-symmetry estimator fits the same data. Declaring the 1–5 scale adds Brennan-Prediger to Conger and Fleiss:
agreement(dat.tenhove2025, estimator = "nt_fiml_cs", categories = 1:5)
#> misskappa: estimator = nt_fiml_cs, weight = quadratic, n = 60
#> estimate se lower upper
#> Conger 0.6845 0.0539 0.5789 0.7902
#> Fleiss 0.6837 0.0542 0.5773 0.7900
#> Brennan-Prediger 0.8120 0.0264 0.7603 0.8637All three coefficients come with sandwich standard errors, recovered from a design where the saturated estimators cannot be computed at all.
How the implementation is validated
The estimators are checked at three levels. The checks live in the package’s test suite and the C++ unit tests, and they run on every change.
Parity with established software
On complete data, the Conger, Fleiss, and Brennan-Prediger estimates and their variances match the irrCAC package (Gwet 2019) on a raw-ratings example and a counts-format example (nominal weights). The irrCAC values are frozen in the test suite, so the check runs without irrCAC installed.
For scored ratings, the normal-theory FIML mean and covariance agree with the saturated estimator in lavaan. This comparison runs whenever lavaan is installed.
Independent reference implementations
The normal-theory FIML fit and its sandwich covariance are compared with a separate pure-R implementation kept in the test suite: an EM for the saturated Gaussian mean and covariance, analytic casewise scores, and a finite-difference observed information. The analytic gradients that map the fitted moments to the coefficients are checked against numerical derivatives.
The C++ unit tests freeze values from the package’s original implementation and check known special cases. Two raters with complete nominal ratings reproduce Cohen’s kappa, perfect agreement gives a coefficient of one, the available-case, IPW, and Gwet estimators coincide on complete data, and every estimated covariance matrix is symmetric and positive semidefinite.
Internal consistency of the inference
Every estimator returns per-subject influence rows fit$psi, and the tests check the identity
\[ \widehat{\mathrm{Cov}}(\widehat\theta) = \frac{1}{n^2}\,\psi^\top\psi \]
for each of them. The equality tests, joint covariances, and custom contrasts all rest on this identity.
What the tests do not cover
Consistency under MCAR and MAR, standard-error calibration, and interval coverage are statistical properties of the estimators, not of the code. They are studied by simulation in the companion methods papers (in preparation).
References
Brennan, R. L., & Prediger, D. J. (1981). Coefficient kappa: Some uses, misuses, and alternatives. Educational and Psychological Measurement, 41(3), 687–699.
Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37–46.
Cohen, J. (1968). Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin, 70(4), 213–220.
Conger, A. J. (1980). Integration and generalization of kappas for multiple raters. Psychological Bulletin, 88(2), 322–328.
Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5), 378–382.
Gwet, K. L. (2019). irrCAC: Computing chance-corrected agreement coefficients among raters. R package.
ten Hove, D., Jorgensen, T. D., & van der Ark, L. A. (2025). How to estimate ICCs for interrater reliability from planned incomplete data. Multivariate Behavioral Research, 60(5). https://doi.org/10.1080/00273171.2025.2507745
