IIS: RI: Small: Nonlinear Dynamical System Theory for Machine Learning

University Of California-Irvine, Irvine CA

Investigators

Abstract

Learning complex statistical models from data is intractable for many models of interest. The PIs are studying a new approach to learning from data that formulates learning as a weakly chaotic nonlinear dynamical system. They show that this dynamical system, which they call ?herding?, combines learning and inference into one tractable forward mapping. They study the abstract mathematical properties of this nonlinear mapping, such as the properties of its attractor set and the topological and metric entropy of the mapping. They then relate these to properties of learning systems. The PIs apply herding systems to a wide range of applications in machine learning. In supervised learning they show that herding suggests a natural extension to the ?voted perceptron algorithm? by including hidden variables. In unsupervised learning, herding is used to train Markov random field models from data. Herding is also extended to Hilbert spaces where it naturally leads to a deterministic sampling algorithm. Due to negative autocorrelations, this ?kernel herding? generates samples that have superior convergence properties than random sampling. They also apply herding to active learning problems. Herding has the potential to radically transform the way we view learning systems. It connects learning to the vast field of nonlinear dynamical systems and chaos theory. As such the impact on machine learning is significant. Scientific results will be disseminated through journal publications and conference proceedings. The PIs also introduce a new course on learning, chaos and fractals to expose students to the intriguing connections between these fields.