An MTurk crisis? Shifts in data quality and the impact on study results
M Chmielewski, SC Kucker - Social Psychological and …, 2020 - journals.sagepub.com
Amazon's Mechanical Turk (MTurk) is arguably one of the most important research tools of
the past decade. The ability to rapidly collect large amounts of high-quality human subjects …
the past decade. The ability to rapidly collect large amounts of high-quality human subjects …
Conducting behavioral research on Amazon's Mechanical Turk
Abstract Amazon's Mechanical Turk is an online labor market where requesters post jobs
and workers choose which jobs to do for pay. The central purpose of this article is to …
and workers choose which jobs to do for pay. The central purpose of this article is to …
Crema-d: Crowd-sourced emotional multimodal actors dataset
H Cao, DG Cooper, MK Keutmann… - IEEE transactions on …, 2014 - ieeexplore.ieee.org
People convey their emotional state in their face and voice. We present an audio-visual
dataset uniquely suited for the study of multi-modal emotion expression and perception. The …
dataset uniquely suited for the study of multi-modal emotion expression and perception. The …
Methods to detect low quality data and its implication for psychological research
EM Buchanan, JE Scofield - Behavior research methods, 2018 - Springer
Web-based data collection methods such as Amazon's Mechanical Turk (AMT) are an
appealing option to recruit participants quickly and cheaply for psychological research …
appealing option to recruit participants quickly and cheaply for psychological research …
Anonymity, intimacy and self-disclosure in social media
Self-disclosure is rewarding and provides significant benefits for individuals, but it also
involves risks, especially in social media settings. We conducted an online experiment to …
involves risks, especially in social media settings. We conducted an online experiment to …
Cognitive biases in crowdsourcing
C Eickhoff - Proceedings of the eleventh ACM international …, 2018 - dl.acm.org
Crowdsourcing has become a popular paradigm in data curation, annotation and evaluation
for many artificial intelligence and information retrieval applications. Considerable efforts …
for many artificial intelligence and information retrieval applications. Considerable efforts …
Shallow pooling for sparse labels
N Arabzadeh, A Vtyurina, X Yan, CLA Clarke - Information Retrieval …, 2022 - Springer
Recent years have seen enormous gains in core information retrieval tasks, including
document and passage ranking. Datasets and leaderboards, and in particular the MS …
document and passage ranking. Datasets and leaderboards, and in particular the MS …
In search of quality in crowdsourcing for search engine evaluation
G Kazai - Advances in Information Retrieval: 33rd European …, 2011 - Springer
Crowdsourcing is increasingly looked upon as a feasible alternative to traditional methods of
gathering relevance labels for the evaluation of search engines, offering a solution to the …
gathering relevance labels for the evaluation of search engines, offering a solution to the …
[HTML][HTML] Resistance to the sharing economy: Why some consumers and providers do not participate in P2P sharing
This study examines the barriers that hinder individuals form participating in the sharing
economy. It analyzes whether non-users' unfavorable perceptions of peer-to-peer (P2P) …
economy. It analyzes whether non-users' unfavorable perceptions of peer-to-peer (P2P) …
Worker types and personality traits in crowdsourcing relevance labels
Crowdsourcing platforms offer unprecedented opportunities for creating evaluation
benchmarks, but suffer from varied output quality from crowd workers who possess different …
benchmarks, but suffer from varied output quality from crowd workers who possess different …