CIF: Small: An Algebraic, Convex, and Scalable Framework for Kernel Learning with Activation Functions

Arizona State University, Scottsdale AZ

Investigators

Abstract

Public interest in machine learning has increased significantly in recent years, with application in a diversity of fields, from medical diagnosis to speech recognition to autonomous driving to advertising. The ability to sustain this interest, however, will depend on whether machine learning algorithms continue to advance in terms of both reliability, scalability, and interpretability. As more data becomes available, Will self-driving cars become safer? Will Siri understand you better? Will doctors be able to better understand the causes and treatments for diseases? While neural networks and deep learning have seen widespread adoption in recent years, the algorithms which underly these methods have not changed substantially in over 20 years. This project, therefore, revisits the fundamental mathematics which underly machine learning algorithms – integrating classical results with the popular neural-network based approaches. This mathematical framework is then used to propose new methods for improving the accuracy of machine learning, for increasing the ability to process large data sets, and for allowing the results of machine learning algorithms to be more readily interpreted in terms of measurable physical quantities. To achieve the goals of accuracy, scalability and interpretability, the project poses an algebraic reformulation of the classical problem of learning the kernel. Specifically, for any given kernel algebra, the positive kernels in that algebra and their associated feature maps may be represented by positive matrices – leading to a convex optimization problem whose solution yields an explicit feature map which may be interpreted in terms of measurable physical quantities. Based on this framework, activation functions are used to define kernel algebras which are universal, yet which are dense in the set of all kernels and whose feature maps mimic those of the neural tangent kernel which defines neural networks – leading to improved accuracy of the algorithms. Next, a saddle-point representation and primal-dual approach is used to convert the kernel learning problem to quadratic programming – resulting in more scalable kernel learning algorithms. Finally, a singular value decomposition of the resulting feature map is obtained by solving an associated partial differential equation. This decomposition is used to identify key features in the data and, furthermore, yields reduced algorithms which scale linearly with the number of samples – implying scalability to datasets with tens of thousands of samples. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.