EAPSI: Audio Attendant: A User Interface for Learning Peripheral Sounds

Williams Kristin, Pittsburgh PA

Investigators

Abstract

The Korean orthography makes clear the relationship between written and spoken language to facilitate phonological decoding of its written form. When a person learns a language, they learn to represent the sounds of the language in meaningful categories. Insofar as dyslexia is a struggle with mapping sounds to their written form, the expression of dyslexia in the Korean language is likely to differ significantly from other languages such as English or Spanish. This project proposes to design and create a user interface accessible in a mobile context that facilitates phonological awareness in the Korean language. The research will be performed in collaboration with Dr. Joonhwan Lee of the Human Computer Interaction and Design Lab at Seoul National University in Korea. The project will inform development of a lightweight dyslexia screener that could be deployed by nonexperts (for free or minimal cost) from around the world to support access to needed resources for a person with dyslexia. The work will advance research areas as language-based assistive technology, human-computer interaction for individuals with cognitive disabilities, and crowdsourcing and will contribute to identifying individuals with dyslexia irrespective of their native language to help make the web accessible to persons with language impairments from around the world. Prior work highlights a role for a system that can help a person implicitly learn sound categories that are specific to a target language, though may not be at the center of a person?s attention or part of their lexicon. This project will investigate how auditory cues can be integrated in a contextually appropriate manner when balancing cues with attending to conversation. Building on Edge, et al., we will extend contextual vocabulary learning of a target language to contextual learning of sound categories. We will determine what sound categories should be acquired by adapting a scoring system of importance that has been found to work well for dynamic visual displays when driving, to an auditory attention service that reflects the learner?s goals and contextual importance. Finally, we will design and create a user interface for learning contextual audio categories based off this prior work to be used in a mobile setting during conversation. This award under the East Asia and Pacific Summer Institutes program supports summer research by a U.S. graduate student and is jointly funded by NSF and the National Research Foundation of Korea.