CHS: Medium: Critical Factors for Automatic Speech Recognition in Supporting Small Group Communication Between People who are Deaf or Hard of Hearing and Hearing Colleagues

Rochester Institute Of Tech, Rochester NY

Investigators

Abstract

To support the employment opportunities of people who are Deaf or Hard of Hearing (DHH), it is important that they be able to communicate effectively with colleagues who can hear. While there are regulations in many educational settings that mandate services be provided by human professionals to support the needs of people who are DHH (e.g., sign-language interpreting or real-time captioning), these services are often not provided in the workplace, where companies may only pay for limited accommodations and where meetings or impromptu chats with a co-worker often arise with little advance notice. As a consequence, DHH employees often have difficulty understanding hearing colleagues, particularly in groups, so many of them prefer to skip meetings and wait for e-mail notes afterward. Mobile "apps" using automatic speech recognition (ASR) to convert audio into text displayed on the screen of a smartphone or tablet have exciting potential for supporting live communication between DHH individuals and hearing co-workers, but even today's state of the art ASR is insufficient to this end due in part to errors that are produced in noisy, real-world settings. This project will explore the user-interface design issues that can increase the benefit to DHH individuals of the sometimes-imperfect captions produced by ASR in small-group meetings with hearing co-workers, and will determine the best ways to measure the usefulness of such technology for members of the DHH community in realistic environments. In addition to creating new scientific knowledge about how people who are DHH make use of emerging speech-recognition technologies to support communication with hearing colleagues in the workplace, project outcomes will provide guidance on the most effective design of mobile apps to enhance this communication so as to transform employment opportunities and independence for these individuals. The findings from this research will also inform other ASR dictation or communication settings, such as how designs can encourage users to speak more clearly for higher ASR accuracy. Research on ASR for live events has primarily focused on lecture transcription, but during a small group meeting participants' behavior adapts dynamically as communication unfolds. The interplay of technology and behavior during interaction often allows users to benefit from ASR even if the text output contains errors. New ASR-for-meetings apps are becoming commercially available, but prior work has revealed that naive designs lead to frustrating interactions; human-computer interaction (HCI) research with DHH users is needed to lay the groundwork for better suited technology. This project will include: surveys of people who are DHH as well as employers and hearing co-workers of DHH individuals; laboratory-based studies with individuals or small-groups to investigate the most effective user-interface designs for mobile apps for this task; participatory design and prototype usability testing of a mobile app based on these studies; and observation of DHH individuals in real-world employment settings using this prototype to determine whether the findings from lab-based studies translate to real-world use. How variations in design parameters, such as latency/speed of captioning, influence the speaking or error-correction behavior of users, and how this can be leveraged to benefit the interaction, will also be explored. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.