SLES: A Theoretical Lens on Generative AI Safety: Near and Long Term

Harvard University, Cambridge MA

Investigators

Abstract

Generative AI technologies like ChatGPT have taken the world by storm with their ability to synthesize strikingly coherent text, code, and more. The pace with which these systems continue to improve in quality and increasingly shape diverse facets of society and industry is remarkable, yet the field's proficiency in controlling and ensuring the reliability of these systems has not quite kept up. These models remain notoriously prone to confidently making factually incorrect yet convincing-sounding statements. Even when they in principle have all of the knowledge that they need to prevent this, the models often still stumble in putting the pieces together. As this technology makes its way into mission-critical contexts like healthcare or policy decisions, it is crucial to avoid such failure modes. This research will develop mathematically rigorous AI deployment methods that come with solid theoretical assurances that the systems will not stray from their intended behavior in this way. The findings of this project will be instrumental in establishing sustainable checks and fail safes so that generative AI technologies can scale in a controlled fashion that is aligned with human interests. The research aims to tackle a mixture of both near-term challenges in safety for generative AI as well as emerging, longer-term ones that will arise as these models grow in their capabilities. For the former, the project will establish mathematical parameters for factuality and non-hallucination in generative models. This encompasses detecting instances when models make factual assertions, calibrating confidence scores for these assertions, reliably attributing these assertions to their sources in the training data, and encouraging models to abstain from generation when faced with sufficiently out-of-distribution input. Another goal is investigating methodologies to elicit and edit knowledge stored in generative models, as well as isolating fundamental barriers to doing so based on tools from fine-grained complexity theory and computational notions of entropy. For safety in the longer-term, the project will examine the feasibility of integrating emergency stop functionality into AI systems based on cryptographic backdoors, as well as implementing "AI arms protocols" based on zero knowledge proofs to publicly certify their safety properties while keeping certain components of these systems private. The research will also rigorously stress-test existing proposals for scalable oversight of AI systems, like natural-language debate and iterated amplification, using techniques from combinatorial game theory and average-case analysis of recursive heuristics. This research is supported by a partnership between the National Science Foundation and Open Philanthropy. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.