CAREER: Inclusive, Private Mobile Input and Interaction Using Lip Reading

University Of California - Merced, Merced CA

Investigators

Abstract

Speech and whisper input on mobile devices can offer fast and seamless hands-free input and interaction to a wide range of users, including people with low vision and blindness. But there are many scenarios where speech and whisper are not viable due to ambient noise or because of privacy and security concerns, or even simply not to disturb other people. A system that understands speech by visually interpreting lip movements, known as image-based lip reading or silent speech, can mitigate many of these challenges. However, silent speech recognition systems are typically slower and more error prone than common speech recognition models, and they may require hardware that is impractical in real-world scenarios. Hence to date this approach has not been investigated as a serious alternative mode of interaction on mobile devices, and it is unknown how best to design the user interface for silent speech or the types of feedback that can enhance its usability. Silent speech has also not been well studied with people without sight. This research will develop an efficient real-time lip reader that uses the front camera of a mobile device to capture the motion of the lips and interprets that into text. A particular focus is on the design of an intuitive user interface that provides a range of visual, auditory, and tactile feedback to facilitate error free text entry. Even broader impacts will derive from providing access to mobile devices to a wider range of users, such as persons with speech disorders or who are mute. Ultimately, project outcomes could be exploited in virtual reality, automotive user interfaces, and many other systems to increase their usability, privacy, security and accessibility. The real-time lip reader will slice and overlap live video feeds from a mobile camera to recognize one phoneme at a time as the user silently speaks by using a deep 3D convolutional neural network (3D-CNN), a recurrent network, and the connectionist temporal classification loss. It will be augmented with a refiner channel that will detect, auto-correct and provide feedback on both character and word-level errors using deep denoising autoencoder (DDA) and custom language models. A range of auditory and tactile feedback will be developed to facilitate error free input and uninterrupted camera view for people with low vision and blindness. The project will also develop multi-modal error correction approaches by exploiting speech, silent speech, and touch interactions. Finally, it will build a silent speech recognition API for the design and development of accessible mobile input and interaction techniques. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.