SHF: Small: Algorithms and Software for Scalable Kernel Methods

University Of Texas At Austin, Austin TX

Investigators

Abstract

Scientists and engineers are increasingly interested in using machine learning methods on huge datasets that cannot be processed on a single workstation. At the same time public and private institutions are making significant investments on high-performance computing (HPC) clusters equipped with thousands of leading edge processors and network connectivity. However, despite the availability of such HPC systems, data analysis tasks are mostly restricted to a single or a few workstations. The reason is that, with few exceptions, existing machine learning software does not scale efficiently on HPC systems. The need to process in-situ large scientific and engineering datasets is not met with current software and significant downsampling is required in order to use existing tools. A serious bottleneck in current artificial intelligence (AI) workflows is the significant cost of training for large scale problems. The slow convergence of existing methods and the large number of calibration hyper-parameters (learning rate, batch size, and other knobs that control the performance of the AI system) make training extremely expensive. Design and analysis of scalable optimization algorithms for faster training, that is the fitting of the machine learning (ML) model parameters to the data, are needed for analytics in real time and at scale, which is the goal of this project. The proposed research will introduce novel numerical methods and parallel algorithms for second-order/Newton methods that will be tailored to machine learning (ML) models and will be many orders of magnitude faster than the existing state of-the-art (first-order methods like steepest descent). The researchers plan to design, analyze, and implement robust approximations for covariance matrices, a class of matrices in AI and computational statistics, used in statistical analysis (e.g., sampling, risk assessment, and uncertainty quantification). The investigators plan to design, analyze, and implement scalable fast algorithms in the context of high-performance computing for the so called nearest-neighbor problem, a particular method in ML, data analysis, and information retrieval. The resulting software library will provide a means for end-to-end tools for discovery and innovation and provide new capabilities in the NSF XSEDE infrastructure project. Along with research activities, an educational and dissemination program is designed to communicate the results of this work to both students and researchers, as well as a more general audience of computational and application scientists. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.