CAREER: Advancing Adversarial Robustness of Natural Language Generation Systems

University Of Texas At Arlington, Arlington TX

Investigators

Abstract

Decision-makers in business, legal, healthcare, and the military use natural language processing systems to obtain insights from vast amounts of data and to make more informed decisions. Recently, natural language generation systems (NLGs) are becoming popular. Examples include question and answer systems and chatbots that are used for advancing public health, and social sensing systems that are used for emergency response and crime prevention. However, there are risks that attackers may be able to manipulate these systems leading to poor outputs and poor decision-making. Robustness to adversaries in deep learning systems has become an active topic in the machine learning and security communities, but the robustness of NLG-based systems is much less studied. This is important to address because there are many differences in the nature of the data and algorithms deep learning and NLG systems employ, as well as the types of tasks they are used for. This project will address these differences through a comprehensive look at the kinds of attacks natural language generation systems are vulnerable to, developing both mathematical models of their vulnerabilities and strategies for reducing them through changes in how NLG systems are designed. This, in turn, will lead to safer, more trustworthy NLG systems and provide a number of educational opportunities for students involved in the research and related classes. The overall goal of the project is to understand NLG systems' attack surface and vulnerabilities and develop novel empirical and theoretical methods for increasing their adversarial robustness. The work will be grounded in two common NLG tasks, summarization and question-answering, and structured around three interconnected aims. The first is developing a framework and proposing novel AI-based optimization methods for examining NLG systems against various attack models. The second is having an in-depth analysis and characterization of vulnerabilities that lead to such attacks. The third is developing a set of defensive methods and tools for enhancing the robustness of NLG systems. This research will be integrated with education and outreach by providing research experiences for for high school, undergraduate, and graduate students, incorporating research results into the course content development and curriculum design, and organizing workshops and competitions to reduce the gap between NLP and cybersecurity programs. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.