Collaborative Research: SaTC: CORE: Medium: Bubble Aid: Assistive AI to Improve the Robustness and Security of Reading Hand-Marked Ballots

William Marsh Rice University, Houston TX

Investigators

Abstract

In the realm of election systems that scan and process hand-marked paper ballots, the most sophisticated ones simply look at the average darkness of marks across a bubble target. This allows traditional paper ballot scanners to miss marks that are insufficiently filled in, or to miscategorize stray marks and scanner noise as filled bubbles. In addition, an inherent security problem is that these systems are not designed to identify potentially fraudulent voting cases where a singular author has filled out ballots for multiple voters. The project’s novelties, responding to these problems, is building Bubble Aid, an Artificial Intelligence (AI) system, aided by data from millions of real ballots that recognizes hand-marked bubble targets more effectively than existing systems. The project's broader significance and importance lies in improving the efficiency and security of election systems that use hand-marked ballots, with wider implications to other applications of hand-marked forms, such as standardized testing. The project is building Bubble Aid as an assistive tool which works as part of a post-election ballot auditing system, which can be used during the "canvass" period before an election is finalized. The goal is to identify the ballots where Bubble Aid disagrees with the official ballot tabulator, and present these ballots to election workers for their ultimate disambiguation. By applying state of the art techniques from the domain of image processing and deep learning, trained on a large corpus of actual ballot marks, Bubble-Aid’s goal is achieving significantly higher accuracy at hand-marked paper ballot scanning than any existing system and also detecting potential fraudulent cases of multiple ballots filled in the same hand. In designing and testing the effectiveness of Bubble Aid, the research team is conducting innovative human-participant experiments for a variety of tasks related to this work. This includes requesting participants to mark test ballots, deliberately injecting noise into them, or filling multiple ballots to emulate the fraudulent scenario, in order to gain additional training data. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.