Wisdom-of-Crowds Approaches for Improving Predictions from Surveys

Santa Fe Institute, Santa Fe NM

Investigators

Abstract

This research project will investigate how survey accuracy can be improved using the wisdom of crowds. Surveys are important tools for making inferences about current attitudes, opinions, and behaviors as well as making forecasts about future trends. However, they are facing threats to their validity that can decrease the accuracy of their inferences. The project will explore the value of two methods based on the wisdom of crowds for improving survey accuracy. One method entails using respondents' knowledge of their social circles, that is, asking respondents to predict the behavior of friends and family. The other method is a scoring procedure that rewards honest and careful answers. By conducting fine-grained tests of the mechanisms underlying these methods, the project will provide insights regarding how to reduce common threats to the validity of surveys. The project also will inform ways to capture the early indications of opinion change relevant for elections and other societal trends. The results could change the way election polls are conducted and improve election predictions. Data from this project will be made publicly available. The results will be disseminated broadly to scientists, practitioners in survey research, polling companies, politicians, policy makers, public educators, journalists, and the general public. This research project will examine how political polling accuracy can be improved by incorporating respondents' knowledge of their social-circles and by scoring respondents' answers with the Bayesian Truth Serum algorithm. Social-circle expectation questions ask respondents how their family and friends will vote. Bayesian Truth Serum is a scoring method that can be used to incentivize and selectively weight participants who provide honest and careful answers. The investigators will assess how the combination of both methods can overcome four recognized threats to polling validity: biased samples, careless responding, socially desirable responding, and social dynamics. A longitudinal survey with a probabilistic national sample will be conducted. The survey will consist of three waves before the 2020 U.S. presidential election and one wave after the election. The investigators will compare traditional survey questions with wisdom-of-crowd methods. Benchmarks will include other concurrent polls and forecasting models and eventually the actual election results and a post-election survey. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.