CDS&E: Efficient and Robust Recurrent Neural Networks

University Of Kentucky Research Foundation, Lexington KY

Investigators

Abstract

Deep neural networks have emerged over the last decade as one of the most powerful machine learning methods. Recurrent neural networks (RNNs) are special neural networks that are designed to efficiently model sequential data such as speech and text data by exploiting temporal connections within a sequence and handling varying sequence lengths in a dataset. While RNN and its variants have found success in many real-world applications, there are various issues that make them difficult to use in practice. This project will systematically address some of these difficulties and develop an efficient and robust RNN. Computer codes derived in this project will be made freely available. The research results will have applications in a variety of areas involving sequential data learning, including computer vision, speech recognition, natural language processing, financial data analysis, and bioinformatics. As in other neural networks, training of RNNs typically involves some variants of gradient descent optimization, which is prone to so-called vanishing or exploding gradient problems. Regularization of RNNs, which refers to techniques used to prevent the model from overfitting the raining data and hence poor generalization to new data, is also challenging. The current preferred RNN architectures such as the Long-Short-Term-Memory networks have highly complex structures with numerous additional interacting elements that are not easy to understand. This project develops an RNN that extends a recent orthogonal/unitary RNNs to more effectively model long and short term dependency of sequential data. Through an indirect parametrization of recurrent matrix, dropout regularization techniques will be developed. The network developed in this project will retain the simplicity and efficiency of basic RNNs but enhance some key capabilities for robust applications. In particular, the project will include a study of applications of RNNs to some bioinformatics problems. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.