CRII: CNS: Exploring Data and Model Sparsity in Deep Learning Systems using Graphs

College Of William And Mary, Williamsburg VA

Investigators

Abstract

Due to the increase in the size of deep learning models, huge computation and memory resources are dedicated for their training, and deployment. To this regard, the proposal treats sparsity as a key primitive to reduce the requirement on the computational and memory resources without sacrificing the accuracy of the models. It seeks to investigate the fundamental understanding of the sparsity in both the data and model at the system-level using graph as a data-model. The preliminary study gathered a few system-level requirements and show that blindly adopting the best practices of sparse linear algebra and graph analytics systems fields, both of which are unrelated to deep learning, lead to many new system-level overheads for deep learning computation. The research proposal will lay down a better sparse data representation to improve the data locality, and will develop theory and systems to solve the workload imbalance problem in the computation involving sparse data and models. The proposal will result in reduction of memory consumption and will achieve faster computation. Both the goals of the proposal will also enable resource-constrained devices to be a part of the deep learning computation ecosystem. Deep learning has enabled use-cases that are being used for diverse societal goals including in the fields of medical science, defense, financial fraud detection, etc. By enabling such use-cases of more diverse datasets and models, both sparse, the proposal seeks to create better impact on the society by achieving faster computation at reduced memory usage, resulting in reduced cost for training and deployment of various deep learning models. The more immediate impact will be in the educational activities in the form of incorporating the developed learning tool(s) for classroom teaching, student projects, outreach to under-represented, minority and female students. Due to the strong interest from the industry in this field, students will be exposed to industry standard tools, as well get trained on end-to-end deep learning systems and data science both-- a key industry requirement that was also identified by the industry partners in an NSF Workshop. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.