CSR: Small: Towards Efficient Deep Inference for Mobile Applications

Worcester Polytechnic Institute, Worcester MA

Investigators

Abstract

An ever-increasing number of mobile applications are using deep learning models to provide novel and useful features, such as language translation and object recognition. These features are supported by passing input data, for example a photo or an audio clip, to complex models in order to generate meaningful output. However, mobile applications that use deep learning models currently need to choose between prediction accuracy and speed at development time. This can lead to poor user experience due to reasons such as running state-of-the-art models on older mobile devices. The proposed MODI (MObile Deep Inference) project outlines new research in designing and implementing a mobile-aware deep inference platform that combines innovations in both algorithm and system optimizations. The proposed work will address mobile deep inference performance problems by enabling flexible, fine-grained model partition and layer-based inference execution, as well as mobile-specific model designs. In addition, MODI enables a scalable mobile deep inference paradigm with efficient model management both on-device and in the cloud. The project will empower deep learning to provide useful features for mobile applications with significantly improved performance. Consequently, this project will open doors to allow running optimized deep learning models on much more resource-constrained devices such as embedded devices. The MODI project can be used as a standalone cloud system or integrated with existing general inference serving platforms by incorporating its mobile-specific optimizations, thereby increasing adoption. The broader impacts of the project will include graduate and undergraduate courses that incorporate research results, outreach to expose undergraduates and K-12 students to research in both computer systems and deep learning. In addition, project related source code and other resources will be released to the research community through the project website at http://tianguo.info/projects/modi.html This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.