CAREER: Learning to Generate American Sign Language Animation through Motion-Capture and Participation of Native ASL Signers

Cuny Queens College, Flushing NY

Investigators

Abstract

American Sign Language (ASL) is the primary means of communication for about 500,000 people in the United States. ASL is a distinct language from English; in fact, a majority of deaf U.S. high school graduates have only a fourth-grade (age 10) English reading level. Consequently, many deaf people find it difficult to read English text on computers, in TV captioning, or in other settings. Software to translate English text into an animation of a human character performing ASL would make more information and services accessible to deaf Americans. Unfortunately, however, essential aspects of ASL are not yet modeled by modern computational linguistic software. Specifically, ASL signers associate entities under discussion with 3D locations around their bodies, and the movement of many types of ASL signs changes based on these locations: pronouns, determiners, many noun phrases, many types of verbs, and others. When do signers associate entities under discussion with locations in space? Where do they position them? How must ASL sign movements be modified based on their arrangement? Creation of robust software to understand or generate ASL requires answers to questions such as these. The PI's goal in this research is to discover techniques for generation of ASL animations that automatically predict when to associate conversation topics with 3D locations, where to place them, and how these locations affect ASL sign movements. To these ends, he will create the first annotated corpus of ASL movement data from native signers (in a motion-capture suit and gloves), annotate this corpus with features relating to the establishment of entity-representing locations in space, use machine learning approaches to analyze when/where these locations are established and how 3D motion paths of signs are parameterized on those locations, incorporate the models into ASL generation software, and recruit native ASL signers to evaluate the 3D animations that result. This work will advance our linguistic knowledge relating to little-understood yet frequent ASL phenomena, and so will lay the foundation for software to produce a huge variety of ASL signs and constructions that are beyond the ability of current techniques to generate. This will in turn lead to ASL generation systems that produce higher quality animations that are more grammatical and understandable to deaf users, which will greatly benefit accessibility applications for deaf users and ASL machine translation. Broader Impact: The ASL motion-capture corpus and an ASL generator that automatically handles spatial phenomena will enable more computational linguistic researchers to study ASL. This research also has applications for sign languages used in other countries (most with similar phenomena), and for the generation of animations of human gestures (for which empirical techniques developed in this work will apply). The PI is committed to finding ways to encourage deaf high school students to pursue science careers, and to creating Ph.D. research opportunities for deaf students. He will give presentations in ASL at local deaf high schools about computing research, make available summer research experiences for deaf high school students (using ASL skills to annotate the corpus, conduct evaluation studies, and inform the Deaf community about computing), recruit native ASL signers as Ph.D. and undergraduate researchers, and create courses on people-focused computer science research and careers (to attract diverse students to the field) and on assistive technology research (to interest and train Ph.D. students). These educational activities will be enabled by the PI's conversational ASL skills, by the research's relevance to deaf students, and by the Queens College's unique proximity to five local high schools for deaf students.