RI: Small: Understanding the Inductive Bias Caused by Invariance and Multi Scale in Neural Networks

University Of Maryland, College Park, College Park MD

Investigators

Abstract

Deep neural networks have had a huge recent impact on the world. They are widely used in systems that understand speech, translate language, and analyze images. In spite of their great impact, researchers still lack a rigorous understanding of many of the basic properties of these networks. As a consequence, new networks are largely designed laboriously, through trial and error. And although extremely effective overall, these systems are sometimes fooled by examples that seem very simple, and similar to other examples that are easily handled. This research aims to provide a better theoretical understanding of an important class of neural networks, called Convolutional Neural Networks (CNNs), which are widely used in understanding images and audio signals. The project focuses on understanding what problems will be easy or difficult for CNNs. This understanding can help us to predict biases in these networks and understand how the design of a network will affect its behavior. The project will provide research opportunities for graduate, undergraduate and high school students, particularly reaching out to students from underrepresented groups. Two key properties that distinguish CNNs from many other approaches to machine learning are their ability to naturally incorporate multiscale analysis and invariance or equivariance. This property has enabled the construction of shift invariant networks that effectively deal with images and signals sampled on grid data, and more recently of networks that handle sets and graphs, incorporating operations that are equivariant to set permutation and graph isomorphism. Multiscale representations naturally arise in these networks through their depth. This research focuses on gaining a better understanding of the role of multiscale, invariance and equivariance in neural networks. It will study how shift invariance and multiscale representations affect the dynamics of neural network training. Our approach will build on recent results showing that massively overparameterized neural networks can be represented as kernel methods. Analyzing the properties of these kernels will help us understand the relationship between a network's architecture and its inductive biases. The insights revealed have the potential to provide a more principled way to control these biases. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.