Collaborative Research: SHF: Medium: Analog EDA-Inspired Methods for Efficient and Robust Neural Network Design

University Of California-San Diego, La Jolla CA

Investigators

Abstract

Deep neural networks have achieved great success in many engineering fields including, but not limited to, image classification, speech recognition, recommendation systems and autonomous driving. However, they suffer from two major challenges. Firstly, many neural network models are not robust, i.e. a neural network could produce inaccurate results when the input data experiences a very small amount of perturbation. Secondly, the huge cost of generating and deploying large-size neural networks limits their applications in resource-constrained platforms (e.g. mobile devices and robots). The research team notices that there is a strong mathematical connection between certain types of neural networks and analog integrated circuits. It is also known that the EDA (electronic design automation) field has 50 years of successful history of modeling, simulating, verifying and optimizing analog integrated circuits. Therefore, this project aims to substantially enrich the algorithms and theoretical understanding of neural networks by leveraging the principled approaches in the EDA community. This research will support the cross-disciplinary development of a diverse cohort of graduate and undergraduate students at the University of California at Santa Barbara, the University of California at San Diego, and the Massachusetts Institute of Technology. Several graduate-level courses on computational methods, data science and artificial intelligence are being created or enriched. The research team willis also collaborating with industry to ensure effective technology transfers. This project focuses on certain types of deep neural networks (e.g., residual neural networks, recurrent neural networks and normalizing flows) that can be described as ordinary differential equations. The technical aims of the project are divided into three thrusts. The first thrust investigates the training and compression algorithms of deep neural networks from circuit simulation and modeling perspectives. Specifically, parallel training algorithms are being developed for neural networks by borrowing the idea from parallel circuit simulation. Hardware-friendly neural-network compression algorithms are being developed from the perspective of circuit model order reduction, thereby enabling energy-efficient and real-time inference of deep neural networks. The second thrust investigates probabilistic and accurate verification techniques for the robustness of deep neural networks from circuit uncertainty quantification perspectives. Specifically, high-confidence and tighter verification bounds are being developed to describe the reachable set of a deep neural network by leveraging the hierarchical and non-Monte-Carlo techniques in analog circuit uncertainty quantification. The third thrust aims to improve the robustness of a deep neural network from the perspective of analog circuit yield optimization. In this final thrust, two ideas are being explored: (1) pre-silicon yield optimization techniques for robust neural network training, and (2) post-silicon self-healing techniques for robustness improvement of a trained neural network. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.