Skip to contents

Estimates Conger, Fleiss, and Brennan-Prediger agreement coefficients (with two raters, Conger is Cohen's kappa) from a subjects-by-raters matrix in which some ratings may be missing. Choose the estimator with estimator:

  • "ipw" (default, recommended): inverse-probability weighting. Consistent under MCAR, also with raters observed at different rates; works with any weight.

  • "cat_fiml": categorical full-information maximum likelihood. Consistent under MAR; feasible for few raters and categories.

  • "pairwise": moment estimator for quadratic weights, MCAR.

  • "nt_fiml": robust normal-theory FIML for quadratic weights, suited to MAR with roughly normal scores. Falls back to "pairwise" when the fit fails; see the section on the fallback.

  • "nt_fiml_cs": "nt_fiml" under the compound-symmetry covariance \(\Sigma = \psi J + e I\); needs only a connected rater co-observation graph, so it handles planned-incomplete designs.

The last three treat ratings as numeric scores and use the quadratic weight.

Usage

agreement(
  x,
  estimator = c("ipw", "cat_fiml", "pairwise", "nt_fiml", "nt_fiml_cs"),
  weight = c("nominal", "linear", "quadratic", "hubert"),
  g = 2L,
  values = NULL,
  categories = NULL,
  em_options = list(),
  fallback = NULL,
  ...
)

Arguments

x

A subjects-by-raters matrix or data frame of ratings, or a subjects-by-raters-by-features array of vector-valued ratings. NA marks missing entries.

estimator

"ipw" (default), "cat_fiml", "pairwise", "nt_fiml", or "nt_fiml_cs".

weight

"nominal" (default), "linear", or "quadratic". The scored estimators always use "quadratic". "hubert" is an all-raters-equal kernel for g > 2. For vector-valued ratings it selects the component loss (Hamming, absolute, or squared).

g

Number of raters the disagreement kernel compares at once. 2 (default) is ordinary kappa. Quadratic agreement does not depend on g.

values

Optional numeric scores, one per category in scale order. Defaults to the categories themselves when numeric, else 1:C.

categories

Optional vector declaring the full rating scale in order, for example 1:5 or c("low", "mid", "high").

em_options

Named list of EM settings for the likelihood estimators: tol and max_iter, and for "cat_fiml" also prune_tol, start_alpha, info_rcond, and flatten. info_rcond is the relative eigenvalue cutoff of the information matrix used for the standard errors. flatten > 0 adds a small Dirichlet pseudo-count that makes the fitted table unique when the likelihood is flat; the estimate barely moves but standard errors become conservative. Settings an estimator does not use are an error.

fallback

For estimator = "nt_fiml" only: "pairwise" (the default) returns the "pairwise" estimate when the fit fails or does not converge; "none" raises the error. See the section on the fallback.

...

Arguments for vector-valued ratings: W (a feature metric for the squared loss), ranges (a features-by-2 matrix of score bounds, for Brennan-Prediger), feature_weights, and loss = "rms". For g > 2: max_chance_tuples (default 5e6). Anything else is an error.

Value

A misskappa_estimate: the estimates, their covariance matrix, the number of subjects n, the rating scale categories, and the per-subject influence rows psi, with vcov = crossprod(psi) / n^2. "cat_fiml" fits also carry two diagnostics per coefficient: null_frac, the share of the coefficient's gradient that lies in directions the likelihood does not identify, and se_cutoff_ratio, the standard error recomputed with info_rcond lowered 100-fold divided by the reported one (values well above 1 mean the standard error leans on weakly identified directions of a sparse table). Methods: print, coef, vcov, confint, as.data.frame, and stats::influence.

Rating scale

Ratings may be numbers, strings, logicals, or factors. The scale defaults to the shared factor levels, or else the sorted observed ratings. Text ratings have no order of their own, so linear and quadratic weights and the scored estimators need categories (or factors with shared levels) for them. Pass categories to declare the full scale including categories nobody used (categories = 1:5 on a 1-5 scale where no one chose 5). The scale sets Brennan-Prediger's chance baseline and the range that scales the linear and quadratic weights. "nt_fiml" and "nt_fiml_cs" report Brennan-Prediger only when categories, values, or factor levels declare the scale; "pairwise" reports Conger and Fleiss only.

Missing data

Subjects with no ratings are dropped. Every rater must be observed, and every rater pair must rate at least one subject together (every g-tuple for g > 2), otherwise the coefficient is not identified and the call errors. "nt_fiml_cs" needs only a connected co-observation graph.

Fallback for "nt_fiml"

The normal-theory likelihood breaks down when the fitted covariance of the ratings is singular or nearly so, for example when two raters agree perfectly on the subjects they share. Kappa itself is still well defined there, but the maximum-likelihood fit and its sandwich standard error are not. This is common in small samples with high agreement and in planned designs where each rater pair shares few subjects. On complete data "nt_fiml" equals "pairwise" and never fails.

By default (fallback = "pairwise"), a failed or non-converged "nt_fiml" fit is replaced by the "pairwise" estimate of the same coefficient, with a warning and the reason recorded in fit$fallback. "pairwise" is the moment estimator that "nt_fiml" reduces to on complete data, so the combination is consistent under MCAR everywhere and keeps "nt_fiml"'s advantage under MAR whenever the fit succeeds. In the simulation study of Moss (in preparation) the combination had smaller bias at n = 40 than either estimator alone, because "nt_fiml" fails selectively on high-agreement samples. Use fallback = "none" to get the error instead.

Generalizations

g > 2 uses g-wise kernels that compare g raters at once (with "ipw" or "cat_fiml"). A subjects-by-raters-by-features array gives vector-valued agreement; see the beyond-pairwise article.

References

Moss, J. (in preparation). Agreement coefficients with missing ratings.

Moss, J. (in preparation). Quadratically weighted agreement coefficients.

See also

agreement_counts() for counts-format data, agreement_test() to compare coefficients, confint.misskappa_estimate().

Examples

# Five raters, three categories, some ratings missing.
agreement(dat.klein2018)
#> misskappa: estimator = ipw, weight = nominal, n = 10
#>                  estimate     se  lower  upper
#> Conger             0.4301 0.1050 0.2244 0.6358
#> Fleiss             0.4054 0.1196 0.1710 0.6399
#> Brennan-Prediger   0.4204 0.1136 0.1978 0.6430

# Ordered categories with linear weights, on a declared 1-4 scale.
agreement(dat.klein2018, weight = "linear", categories = 1:4)
#> misskappa: estimator = ipw, weight = linear, n = 10
#>                  estimate     se   lower  upper
#> Conger             0.3398 0.1389  0.0675 0.6121
#> Fleiss             0.3026 0.1624 -0.0157 0.6210
#> Brennan-Prediger   0.4764 0.1269  0.2278 0.7251

# Quadratic agreement on a 1-5 scale under MAR, normal theory.
agreement(dat.zapf2016, estimator = "nt_fiml", categories = 1:5)
#> misskappa: estimator = nt_fiml, weight = quadratic, n = 50
#>                  estimate     se  lower  upper
#> Conger             0.8985 0.0278 0.8439 0.9530
#> Fleiss             0.8984 0.0279 0.8437 0.9530
#> Brennan-Prediger   0.8675 0.0313 0.8062 0.9288

# A planned-incomplete design: some rater pairs never meet, so only the
# compound-symmetry estimator is identified.
agreement(dat.tenhove2025, estimator = "nt_fiml_cs")
#> misskappa: estimator = nt_fiml_cs, weight = quadratic, n = 60
#>        estimate     se  lower  upper
#> Conger   0.6845 0.0539 0.5789 0.7902
#> Fleiss   0.6837 0.0542 0.5773 0.7900