Agreement study from McDuff & Girard (2019) in long format: MTurk judges rate images of people smiling. The design is large and incomplete — many items and judges, and no judge rates every image — which makes it a useful missing-data example. There are 273 items, 121 judges, and 1365 rows.
Format
A 1365 x 4 data frame with one row per (item, judge) rating:
- rating_positive
numeric; rated positivity of the image.
- rating_smile
numeric; rated presence of a smile in the image.
- item
integer; the image rated.
- judge
integer; the MTurk judge giving the rating.
Source
The file 04_MTurk.csv at https://osf.io/n4grd/.
References
McDuff, D., & Girard, J. M. (2019). Democratizing Psychological Insights from Analysis of Nonverbal Behavior. 2019 8th International Conference on Affective Computing and Intelligent Interaction (ACII), 220-226. doi:10.1109/ACII.2019.8925503
Examples
# Each image is rated on a 1-6 smile scale by 5 of the 121 MTurk judges, and
# almost no two judges share images, so the judges are not individually
# identified -- this is a crowdsourced design with exchangeable raters.
# Reshape to an (items x judges) matrix, collapse to per-image category
# counts with ratings_to_counts(), then estimate the scored (quadratically
# weighted) agreement.
smile <- with(dat.mcduff2019,
tapply(rating_smile, list(item, judge), function(z) z[1]))
smile <- matrix(as.numeric(smile), nrow = nrow(smile))
kappa_counts(ratings_to_counts(smile), weight = "quadratic")
#> misskappa: estimator=fleiss_cuzick, weight=quadratic
#> estimate se lower upper
#> Fleiss 0.6421 0.0237 0.5958 0.6885
#> Brennan-Prediger 0.6566 0.0255 0.6066 0.7067