CRII: HCC: American Sign Language-enabled Digital Assistants: Models and Metrics for User Satisfaction

Gallaudet University, Washington DC

Investigators

Abstract

Intelligent personal assistants (IPAs) that allow people to interact with computers using natural language are becoming more and more common. These are typically controlled by voice interaction; however, not everyone uses their voice for real-time communication, including many Deaf people who use American Sign Language (ASL) as their primary language for communicating with others. This project's goal is to develop principled methods for studying and designing ASL-based IPAs that allow Deaf ASL users to interact with them through sign. Because computers have trouble recognizing unconstrained ASL communication, the project team will conduct a set of experiments with Deaf users to understand how well interfaces that recognize smaller subsets of ASL can serve their needs, and how much recognition errors affect their utility. The experiments will result in a set of guidelines for how ASL-based IPAs might be designed and implemented, reducing accessibility barriers for those who use ASL. The project will use a Wizard-of-Oz methodology, in which the experimental team crafts responses to simulate computational tasks that are hard to implement, to study different levels of sign language understanding abilities in an IPA. The first research aim will investigate the potential of an IPA interface that uses only single-sign recognition, with a vocabulary of approximately 100 signs that are synecdoches of common IPA voice query phrases. The second research aim will investigate user satisfaction when using a larger, but still limited, ASL vocabulary set to query IPAs. In this experiment, users will use more natural ASL phrases instead of single signs, but the simulated IPA will have a limited understanding and perform accordingly. A third research aim will vary the level of simulated ASL recognition accuracy in the IPA and investigate the level of accuracy needed for user satisfaction in the context of IPAs that understand ASL but make errors. Together, these Deaf-led human-centered experiments will engage with the Deaf community, empirically investigate criteria for successful ASL IPA interaction and provide foundational research for future ASL-based systems. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.