Citizen Science EAGER: Quantifying Uncertainty in Crowd Response for Reliable Wind Hazard and Damage Assessment

University Of Illinois At Urbana-Champaign, Urbana IL

Investigators

Abstract

Damage to infrastructure arising from windstorms exceeds damage from any other natural hazard in the U.S. The highly variable nature of wind loadings on buildings during a windstorm, however, means that accurate characterization at the damage location may not be captured by current measurement networks. Ubiquitous smartphone and internet availability, widespread use and rapid dissemination of social media, the power of crowds engaged in scientific endeavors, and the public's awareness of vulnerabilities point to a paradigm shift in sensing hazards in general. In the case of windstorm damage, on-the-ground data retrieved and shared by Citizen Science public participation may provide windstorm data previously unavailable. The primary focus of this EArly-concept Grant for Exploratory Research (EAGER) award is to study "human-sensor" data collected through Amazon Mechanical Turk-- a crowd sourcing application. Volunteers will be shown images and related data from actual windstorms and asked to characterize the damage and their confidence in their assessments. These data will be used to design a crowd sourcing algorithm that will enable robust Citizen Science public participation in the rapid identification of damage areas to help decision makers to allocate resources for damage response and recovery efforts and for targeted damage assessments, which can help to improve the design of buildings in regions susceptible to intense windstorms. This project will address two key questions: How can one quantify the confidence in crowdsourced damage assessment? How can one design a tool for more reliable crowdsourcing given unreliable participants? Researchers will initially compile a "validation set" of data that includes imagery, damage states and wind speed estimates for approximately 8,000 structures that were affected by the Joplin, MO tornado. The data set will be used to create a reliable crowdsourced image classification scheme in the form of online questionnaires for the public. The questionnaires will be tested by collecting assessment reports from participants on Amazon Mechanical Turk. These reports will inform development of an uncertainty model for participant reliability, which forms the basis for a coding-theoretic crowdsourcing algorithm that is robust to uncertainties due to unreliable participants. This algorithm will be tested against a separate dataset to compare the researchers' approach with one that doesn't control for participant unreliability. The final research results will be shared with NOAA for use in training surveyors to assess wind damage and to provide tutorials for the public.