ITR: Multimodal Learning for Assistive Aids

Massachusetts Institute Of Technology, Cambridge MA

Investigators

Abstract

This is the first year funding of a three-year continuing award. Multimodal technologies and machine learning have the potential to open doors for individuals with disabilities, by enabling systems to provide interfaces which allow people to express themselves through a variety of modalities which suit their capabilities. But individual differences among potential users make it difficult to design a single interface that works for everyone. Adaptive systems which tune to a user's idiosyncratic abilities and preferences can eliminate the need for users to conform to fixed interface protocols. Previously, the P1 has achieved promising results both in developing models of learning from multimodal input, and in discovering acoustic features that carry information in severely impaired speech. In this project the PI will develop a framework for adaptive interfaces by integrating these threads of research. Multimodal learning will provide interfaces with a core adaptive engine that can detect and statistically model salient inter-modal patterns. Acoustic analysis of impaired speech will provide one of many modes of input which an individual with disabilities might use to express him or herself. By combining multiple modes of sensing with learning, an interface can be trained to respond to an individual's unique expressive behaviors (speech., gestures) and translate them into appropriate machine actions. The P1 will implement two assistive communication aids to test our. these ideas. The first prototype will learn to translate unintelligible spoken phrases into clear synthetic speech. To do so, it will learn consistent acoustic features of the user's voice that can reliably be mapped onto machine actions. The second system will dynamically adjust the display of a communication aid by predicting words and symbols that the user would most likely select. Predictions are based on observations of patterns of behavior exhibited by the user in past communication interactions. Unlike currently available word prediction systems, this interface will take into account the topic of conversation by analyzing the speech of the user's communication partner using speech recognition and topic identification technologies. Both interfaces will undergo usability testing in hospitals and clinics in Boston and Toronto. Based on these efforts, the P1 will derive a set of design principles for using adaptive elements in assistive communication aids and man-computer interfaces.