CRII: SaTC: RUI: A Cross-Verification Approach for Identifying Tampered Audio

Harvey Mudd College, Claremont CA

Investigators

Abstract

Recent advances in technology have made it possible to create fake videos of celebrities doing or saying things they did not do or say. There is concern that this technology may be used to influence elections by defaming candidates or to instigate civil unrest through false statements by public officials. This project focuses on protecting world leaders from such fake impersonations. It approaches the problem from a history-centric perspective by asking the question, “Is this video a historically verifiable event?” In the same way that a historian tests a historical claim by cross-checking against other primary sources, we can test the authenticity of an audiovisual recording by cross-checking against other primary sources of audiovisual information. This project develops such a cross-verification approach for identifying fake or tampered audio content. The award will support undergraduate research at a highly diverse liberal arts college. This project investigates audio cross-verification in two scenarios. In the first scenario, the goal is to cross-verify a query with a trusted source recording, which is assumed to be reliable. In order to identify tampering via insertion, deletion, or modification, the alignment between the query and source recording can be computed using dynamic time warping (DTW). The key contribution in this scenario is developing an understanding of how to incorporate DTW into an end-to-end learning system using an appropriate smooth relaxation of DTW during training. In the second scenario, the goal is to cross-verify a query with untrusted source recordings, which may themselves be tampered. The key contribution in the second scenario is to extend existing alignment techniques to jointly align a collection of recordings in the presence of malicious tampering, and to reconstruct the audio ground truth. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.