EAGER: Exploring Motivated Misreporting of Crowd Workers for Developing Data Quality Assurance Best Practices for Using Crowdsourcing Platforms in Engineering Research

Research Triangle Institute, Durham NC

Investigators

Abstract

Researchers increasingly use online platforms to tap into the knowledge of crowds. They divide complex scientific and engineering work into small, manageable tasks that crowd workers can complete. This method is quicker and cheaper than using trained scientific researchers, and has been used to disseminate emergency information, monitor hazardous events, and improve performance of information systems or other engineering practices. However, the data from such online platforms may be subject to various factors which can reduce overall quality. Moreover, the internal quality control mechanisms, if any, offered by common online crowdsourcing platforms may not let researchers assess data quality appropriately. This EArly-concept Grant for Exploratory Research (EAGER) project will draw on social science theory to investigate the tendency of crowd workers to cut corners to reduce their workload while still collecting incentives for the task completion. The immediate impact of this cross-disciplinary research is to inform quality assurance practices for managing crowdsourcing tasks of various types and at different scales and paces. The results will improve researchers' use of crowdsourcing to engage the general public in rigorous scientific and engineering research. The research outcome may further enhance crowdsourcing applications in emergency response, post-disaster damage assessment, and other social contexts where reliable data quality is required for successful engineering practices. Guided by the research on motivated misreporting in social science methodology, this study will conduct experiments to investigate whether online research participants tend to shirk their assigned tasks to reduce their workload while still collecting incentives for completing the task. The PIs will study three common crowdsourcing tasks: coding of satellite images after a natural disaster, responding to surveys, and classifying sentiments of online social media content. The experiment will recruit workers from popular crowdsourcing platforms, panelists from commercial online survey panels, and citizen scientists from online volunteering sites. The experiment will explore the following questions: (1) are crowd workers likely to engage in motivated misreporting when they complete tasks online If so, do the patterns of misreporting vary by the different incentive schemes provided by the platforms? And (2) do the patterns of misreporting vary by different task types The results will inform the development of best practices and quality assurance procedures for researchers interested in incorporating collective intelligence to improve the system for massive online information analysis in engineering, computer, and social sciences.