CRII: III: Trustworthy Diffusion Models

University Of North Carolina At Charlotte, Charlotte NC

Investigators

Abstract

The rapid development and attention towards generative artificial intelligence (AI) makes their trustworthiness a critical issue of societal importance. Diffusion models (DMs) are large generative AI models which can generate high quality images from text instructions. There are concerns about the trustworthiness of DMs in terms of privacy, fairness and explainability. It may over-memorize the training data, which causes vulnerabilities to privacy leakages. Without proper guidance, it may inherent social bias from the training data and generate harmful images towards unprivileged groups. It cannot explain why or how the images are generated based on the instructions. This project evaluates the trustworthiness of DMs and provides advanced solutions to address these issues. The results of this project can benefit decision-makers and practitioners in different areas such as health care, media, law, and education to adopt generative models to assist content creation and daily productivity. This project extends the current techniques in Diffusion Models focusing on a single aspect of trustworthiness to achieve multi-desiderata simultaneously including privacy, fairness, and explainability. This project first implements differential private DMs to defend against privacy attacks. It then introduces fair training in private DMs to avoid spurious correlation and stereotyping in generated content. In addition, it explores faithful explanations of DMs with the assistance of attention mechanisms. Most importantly, the projects evaluate the trade-off between these trustworthiness properties and generative quality in a joint trustworthy DM framework. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.