EAGER: DCL: SaTC: Enabling Interdisciplinary Collaboration: Improving Human Discernment of Audio Deepfakes via Multi-level Information Augmentation

University Of Maryland Baltimore County, Baltimore MD

Investigators

Abstract

This project increases listeners’ discernment of audio deepfakes through augmentation of information, both technological and sociolinguistic. This project establishes an innovative pathway for collaborative research across sociolinguistics, human centered analytics, and data science and lays the groundwork for future analyses of deepfakes that are broadly relevant across disciplines, informed by human behavioral perspectives. The project will address the societal challenge of misinformation by generating insights that can increase the ability of listeners – particularly college students, whose lives are indelibly shaped by technology – to evaluate the veracity and authenticity of information online. The project's broader significance is to address the societal challenge of misinformation by generating insights that can help empower listeners to make decisions about how to evaluate the veracity and authenticity of information they encounter online. The project improves understanding and modeling of how deepfakes are involved in spreading misinformation and tracking how language technology is adapted for social harm and/or used in unethical ways. The proposed work will increase listeners’ discernment of audio deepfakes through augmentation of information that draws upon integrated interdisciplinary knowledge and advances data augmentation as an important tool for deepfake detection. The objectives of the project are to: (1) Study and evaluate listener perceptions of audio deepfakes that have been created with varying degrees of linguistic complexity; (2) Study and evaluate the efficacy of training sessions that increase listeners’ sociolinguistic perceptual ability and improve their ability to discern deepfake audio content; (3) Augment the audio deepfake discernment via multi-level temporal and linguistic signatures, informed by training and linguistic labeling; (4) Evaluate the impact of augmented signature information on listener perceptions of audio deepfakes; (5) Create open-access online modules and materials with social science and data science student involvement to improve listeners’ discernment of audio cues on a wider public scale. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.