Estimates Conger / Fleiss / Brennan-Prediger weighted agreement coefficients for raw ratings with missing entries. Pick the estimator with `estimator`:
`"ipw"`: inverse-probability-weighted moment estimator for categorical ratings under MCAR, consistent under non-exchangeable raters with rater-varying observation rates.
`"cat_fiml"`: saturated-multinomial full-information maximum likelihood for categorical ratings under ignorable missingness.
`"pairwise"`: pairwise-available moment estimator for the quadratically weighted (scored) coefficient under MCAR.
`"pairwise_cs"`: the MCAR moment estimator under the compound-symmetry covariance model \(\Sigma = \psi J + e I\) (fixed raters with free means, the distribution-free companion to `"nt_fiml_cs"`, needing only a connected rater co-observation graph).
`"nt_fiml"`: robust normal-theory FIML (saturated covariance) for the quadratically weighted coefficient under ignorable missingness.
`"nt_fiml_cs"`: the same FIML under the compound-symmetry covariance model \(\Sigma = \psi J + e I\) (fixed raters, needs only a connected rater co-observation graph).
`"ipw"` and `"cat_fiml"` take integer category codes and the nominal, linear, or quadratic `weight`. `"pairwise"`, `"pairwise_cs"`, `"nt_fiml"`, and `"nt_fiml_cs"` are quadratic by construction: they treat `x` as numeric scores (pass scores directly, or integer codes with `values`) and require `weight = "quadratic"`.
Two generalizations are reached through the same call. Set `g > 2` for the Frechet / Hubert g-wise (multirater) kernels, which report the Conger-type (distinct rater combinations) and Fleiss-type coefficients. Pass a subjects-by-raters-by-features array to score vector-valued ratings. With `"pairwise"` or `"ipw"` and no feature metric the loss is component-separable (diagonal feature weights via `feature_weights` / `loss`). Pass a full feature metric `W` (or use `"nt_fiml"` / `"nt_fiml_cs"`) for the squared vector loss `(x - y)' W (x - y)`: `"pairwise"` is the moment estimator and `"nt_fiml"` / `"nt_fiml_cs"` the saturated / block compound-symmetry normal-FIML (`Sigma = J_R %x% Psi + I_R %x% E`). See the arguments for the estimator / weight combinations supported in each mode.
For scalar fixed-rater coefficients with missing entries, the observed-data pattern must identify the requested functional. For ordinary pairwise kappa (`g = 2`, and all quadratic requests), every rater must be observed at least once and every rater pair must be jointly observed by at least one subject. For non-quadratic `g > 2`, every requested rater g-tuple must be jointly observed by at least one subject. Violations are reported as non-identification before the estimator runs.
Arguments
- x
A subjects-by-raters matrix or data frame of scalar ratings, or a subjects-by-raters-by-features 3-D array of vector-valued ratings (the array form is auto-detected). Integer category codes for `"ipw"` / `"cat_fiml"`. Numeric scores for `"pairwise"` / `"nt_fiml"`. `NA` marks missing entries.
- estimator
One of `"ipw"`, `"cat_fiml"`, `"pairwise"`, `"pairwise_cs"`, `"nt_fiml"`, or `"nt_fiml_cs"` (the `_cs` names are the compound-symmetry covariance variants of `"pairwise"` and `"nt_fiml"`). Vector-valued ratings support `"pairwise"` and `"ipw"` (component-separable) and, with a full feature metric `W`, `"pairwise"`, `"nt_fiml"`, and `"nt_fiml_cs"` (full-`W` squared loss). Continuous g-wise kernels (`weight = "linear"`, `g > 2`) support `"pairwise"` and `"ipw"` only.
- weight
Weighting scheme for `"ipw"` / `"cat_fiml"`: `"nominal"` (the default), `"linear"`, or `"quadratic"`. `"pairwise"` / `"pairwise_cs"` / `"nt_fiml"` / `"nt_fiml_cs"` require `"quadratic"`. For `g > 2` only `"nominal"`, `"linear"`, `"hubert"`, and `"quadratic"` are defined, where `"hubert"` is the all-raters-equal multirater kernel (g > 2 only). For vector-valued ratings `weight` selects the component loss (`"nominal"` -> Hamming, `"linear"` -> L1, `"quadratic"` -> squared).
- g
Arity of the multirater disagreement kernel. `g = 2` (the default) is ordinary pairwise kappa. `g > 2` uses the Frechet / Hubert g-wise family. `weight = "quadratic"` is g-invariant, so `g` is ignored there and the cheap closed form is used. Ignored for vector-valued ratings.
- values
Optional numeric vector of category scores used by the metric weightings (and by the quadratic / continuous g-wise estimators to map integer codes to scores). Defaults to the sorted unique observed categories.
- em_options
Named list of EM options for the likelihood estimators (`"cat_fiml"`, `"nt_fiml"`). For `"cat_fiml"`: `tol`, `max_iter`, `prune_tol`, `start_alpha`, `info_rcond`, and `flatten` (total Dirichlet pseudo-mass spread over the complete pattern table. Any positive value selects the unique interior posterior mode when the saturated likelihood is flat, shrinking the fitted table toward uniform with weight `flatten / (n + flatten)`. `0`, the default, is strict ML). Flattening is a uniqueness device that leaves the point estimate essentially unchanged but makes the reported standard errors conservative (roughly 50% too wide in calibration). For `"nt_fiml"` / `"nt_fiml_cs"`: `tol` and `max_iter` (the covariance structure is selected by the estimator name). `covariance` and `mean` are deprecated here (use `estimator = "nt_fiml_cs"`, the surface is mean-free) and warn if supplied. Pass any subset.
- ...
Mode-specific extras. `feature_weights` and `loss` (e.g. `"rms"`) for component-separable vector-valued ratings. `W`, a full features-by-features symmetric positive-semidefinite feature metric, for the full-`W` squared vector loss (`"pairwise"` / `"nt_fiml"` / `"nt_fiml_cs"`, defaulting to the identity). `ranges`, a features-by-2 matrix of `(lo, hi)` score bounds that fixes the Brennan-Prediger chance baseline for the full-`W` `"nt_fiml"` / `"nt_fiml_cs"` estimators instead of estimating it from the observed range (see [kappa_vector_quadratic()]). Finally, `max_chance_tuples` (cap on the number of chance tuples enumerated) for `g > 2`.
Value
An object of class `misskappa_estimate` carrying the named coefficient estimates and their asymptotic covariance matrix. Methods: `print`, `coef`, `vcov`, `confint`, `as.data.frame`, and `stats::influence`. The object also carries a `psi` component (the n-by-K matrix of per-subject influence functions, satisfying `vcov == crossprod(psi) / n^2`) for power users who want to build their own contrasts or joint tests.
Details
The public R API deliberately exposes only the main weighting schemes used in current agreement work: `"nominal"`, `"linear"`, and `"quadratic"` (plus the `"hubert"` g-wise kernel for `g > 2`). Older Krippendorff / irrCAC-style categorical schemes (`"ordinal"`, `"radical"`, `"ratio"`, `"circular"`, `"bipolar"`) remain in the internal C++ layer and Rcpp glue for validation and parity work, but are unsupported and not part of the recommended R interface.
References
Moss, J. (in preparation). *Agreement coefficients with missing ratings.* Working paper (kappa-missing); citation to be updated on preprint and publication.
Moss, J. (in preparation). *Quadratically weighted agreement coefficients.* Working paper (quadratic); citation to be updated on preprint and publication.
Krippendorff, K. (2011). Computing Krippendorff's alpha-reliability.
Gwet, K. L. (2019). irrCAC: Computing chance-corrected agreement coefficients among raters.
Examples
# Categorical ratings with missing entries (Gwet 2014): the
# inverse-probability-weighted estimator under MCAR, with linear weights.
kappa(dat.gwet2014, estimator = "ipw", weight = "linear")
#> Warning: rater pair(s) rater4-rater5 co-observed by only one subject; the corresponding pairwise covariance is degenerate and the standard error unreliable.
#> misskappa: estimator=ipw, weight=linear
#> estimate se lower upper
#> Conger 0.5879 0.0772 0.4365 0.7393
#> Fleiss 0.5858 0.0774 0.4341 0.7375
#> Brennan-Prediger 0.6397 0.0767 0.4894 0.7900
# Scored coefficient (quadratic loss) via the pairwise-available moment
# estimator. Treats the columns as numeric scores.
kappa(dat.zapf2016, estimator = "pairwise")
#> misskappa: estimator=pairwise, weight=quadratic
#> estimate se lower upper
#> Conger 0.8985 0.0278 0.8439 0.953
#> Fleiss 0.8984 0.0279 0.8437 0.953
# Compound-symmetry covariance variant of the normal-theory FIML: the
# fixed-rater variance-component model. Consistent and slightly more
# efficient than "nt_fiml" when the structure holds, and identified by a
# connected (not necessarily complete) rater co-observation graph.
kappa(dat.zapf2016, estimator = "nt_fiml_cs")
#> misskappa: estimator=nt_fiml_cs, weight=quadratic
#> estimate se lower upper
#> Conger 0.8985 0.0278 0.8439 0.9530
#> Fleiss 0.8984 0.0279 0.8437 0.9530
#> Brennan-Prediger 0.8012 0.0469 0.7093 0.8932
# The distribution-free MCAR moment companion to "nt_fiml_cs".
kappa(dat.zapf2016, estimator = "pairwise_cs")
#> misskappa: estimator=pairwise_cs, weight=quadratic
#> estimate se lower upper
#> Conger 0.9002 0.0278 0.8458 0.9547
#> Fleiss 0.9002 0.0278 0.8457 0.9547
#> Brennan-Prediger 0.8051 0.0469 0.7132 0.8970
# Planned-incomplete design (synthetic): some rater pairs are never
# co-observed, so the saturated estimators are not identified, but the
# compound-symmetry estimator needs only a connected co-observation graph.
kappa(dat.tenhove2025, estimator = "nt_fiml_cs")
#> misskappa: estimator=nt_fiml_cs, weight=quadratic
#> estimate se lower upper
#> Conger 0.6845 0.0539 0.5789 0.7902
#> Fleiss 0.6837 0.0542 0.5773 0.7900
#> Brennan-Prediger 0.7180 0.0395 0.6405 0.7955
# g-wise (multirater) nominal kappa over triples of raters.
kappa(dat.gwet2014, estimator = "ipw", weight = "nominal", g = 3)
#> Warning: rater 3-tuple(s) rater1-rater4-rater5, rater2-rater4-rater5, rater3-rater4-rater5 co-observed by only one subject; the corresponding observed-disagreement term is degenerate and the standard error unreliable.
#> misskappa: estimator=ipw, weight=nominal, g=3
#> estimate se lower upper
#> Conger 0.4369 0.0850 0.2703 0.6034
#> Fleiss 0.4308 0.0861 0.2620 0.5997
# Vector-valued ratings: a subjects-by-raters-by-features array. Each of the
# four expert observers classifies crackles at six chest sites, so a rating
# is a six-vector. The default Hamming loss counts disagreeing sites.
kappa(dat.vanbelle2019[, 1:4, ], estimator = "ipw")
#> misskappa: estimator=ipw, weight=hamming
#> estimate se lower upper
#> Conger 0.5192 0.0794 0.3635 0.6748
#> Fleiss 0.5150 0.0813 0.3557 0.6743
# Full feature-metric quadratic vector agreement under the block
# compound-symmetry normal FIML, with a feature metric W over the six sites.
sites <- dat.vanbelle2019[, 1:4, ]
kappa(sites, estimator = "nt_fiml_cs", W = diag(dim(sites)[3]))
#> misskappa: estimator=nt_fiml_cs, weight=quadratic
#> estimate se lower upper
#> Conger 0.5192 0.0794 0.3635 0.6748
#> Fleiss 0.5150 0.0813 0.3557 0.6743
#> Brennan-Prediger 0.1333 0.1465 -0.1539 0.4205