I-Corps: End-User Trained Augmentative/Alternative Communication Tablet for Speech-Impaired Patients

Western Michigan University, Kalamazoo MI

Investigators

Abstract

The broader impact/commercial potential of this I-Corps project stems from exploiting the powerful image and pattern recognition abilities of artificial intelligence (AI) to create a communication assistance device that does not require the user to adapt to the inherent limitations of a new technology. Instead, the device adapts to the user, within whatever limitations they experience, for near-immediate uptake. Consequently, the significant barriers associated with introducing unfamiliar technology are removed, and the user - regardless of the underlying physical impairment to verbal communication - can interact with others, their environment (through linked internet-of-things enabled smart devices) and equipment or instrumentation interfaces. Particulary for those with a condition permanently limiting verbal communication, a practical, adaptive and viable approach to interacting with others can reduce the reliance on caregivers, and enable them to more fully integrate in their environment. A more detailed understanding of the practical obstacles to introducing new communication devices will translate to related fields as well, including situations characterizing medical, hazardous disposal, and harsh environments. This I-Corps project builds upon the successful demonstration of machine-learning ("artificial intelligence, or AI") image processing techniques to create a viable alternative method for nonverbal communication. The input device is a conventional tablet or smartphone, and the typical user will interact with a single finger, although alternatives are conceivable. The project originates from a study focused on development of next-generation, environmentally-friendly, earth-abundant element semiconductors, in which large amounts of video data needed to be processed in an automated fashion. The underlying framework for the corresponding tool that was developed was adapted to this new, but not unrelated, purpose, so that the communication device adapts to the user and the user's limitations, and not the other way around. We have established that is is feasible to process user-defined abstract input essentially in real time by capitalizing on current generation hardware adapted for AI-based applications. The user trains a neural network, which becomes increasingly more accurate with use. The project will identify meaningful success metrics for target markets so that the corresponding minimum viable product can be compared to existing and competing technologies, as well as identify the primary barriers to technology uptake that can come into play. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.