Skip to contents

Estimates Fleiss and Brennan-Prediger coefficients when each row is a subject and each column a category, with x[i, k] the number of raters who put subject i in category k. Rows may have different totals, which is how missing ratings show up in this format. Raters are anonymous, so they are treated as exchangeable and Conger is not reported. For rater-identified data use agreement(); ratings_to_counts() converts it to this format.

  • "pairwise" (default, recommended): the moment estimator of Fleiss and Cuzick (1979), extended to linear and quadratic weights. It averages agreement over the rater pairs within each subject, the counts-format analogue of "pairwise" in agreement(). Consistent under MCAR, any weight.

  • "cat_fiml": full-information maximum likelihood over the category compositions, consistent under MAR. It is the "cat_fiml" of agreement() with the raters constrained to be exchangeable (the same likelihood), which is why it shares the name.

  • "nt_fiml_cs": normal-theory compound-symmetry FIML, quadratic weight only.

Rows with fewer than two ratings carry no agreement information; at least one row needs two or more.

Usage

agreement_counts(
  x,
  estimator = c("pairwise", "cat_fiml", "nt_fiml_cs"),
  weight = c("nominal", "linear", "quadratic"),
  values = NULL,
  r_total = NULL,
  em_options = list()
)

Arguments

x

A subjects-by-categories matrix of non-negative counts.

estimator

"pairwise" (default), "cat_fiml", or "nt_fiml_cs".

weight

"nominal" (default), "linear", or "quadratic". "nt_fiml_cs" always uses "quadratic".

values

Optional numeric category scores for the linear and quadratic weights. Defaults to 1:C.

r_total

For "cat_fiml": the number of raters each subject would have with no missing ratings. Defaults to the largest row total.

em_options

Named list of EM settings for the likelihood estimators: tol and max_iter, and for "cat_fiml" also prune_tol, start_alpha, and info_rcond.

Value

A misskappa_estimate with Fleiss and Brennan-Prediger; see agreement().

References

Fleiss, J. L., & Cuzick, J. (1979). The reliability of dichotomous judgments: Unequal numbers of judges per subject. Applied Psychological Measurement, 3(4), 537-542.

Moss, J. (in preparation). Quadratically weighted agreement coefficients.

Examples

# Fleiss (1971): 30 subjects, 6 raters, 5 diagnostic categories.
agreement_counts(dat.fleiss1971)
#> misskappa: estimator = pairwise, weight = nominal, n = 30
#>                  estimate     se  lower  upper
#> Fleiss             0.4302 0.0533 0.3258 0.5347
#> Brennan-Prediger   0.4444 0.0542 0.3382 0.5507

# Ordered categories with quadratic weights.
agreement_counts(dat.fleiss1971, weight = "quadratic")
#> misskappa: estimator = pairwise, weight = quadratic, n = 30
#>                  estimate     se  lower  upper
#> Fleiss             0.2841 0.1093 0.0698 0.4983
#> Brennan-Prediger   0.3339 0.1019 0.1342 0.5336