Essential Tools for Computational Research on Visual-Gestural Language

Trustees Of Boston University, Boston

Investigators

Abstract

This is the first-year funding of a three year continuing award. Research on recognition and generation of signed languages and the gestural component of spoken languages has been held back by the unavailability of large-scale linguistically annotated corpora of the kind that led to significant advances in the area of spoken language. A major obstacle to the production of such corpora has been the lack of computational tools to assist in efficient analysis and transcription of visual language data. In this project the PI and her team will develop the tools and techniques necessary to support efficient transcription of finely detailed phonological information and its integration with information provided by computational algorithms, so as to enable the creation of large-scale corpora annotated at the level of granularity essential for computer science research. Machine vision-based algorithms for semi-automation of several aspects of the transcription process will also be developed. This research will result in the production and public dissemination of corpora that will include fine-grained linguistic annotations of ASL video data. The availability of these corpora and the refinement of computational tools for analyzing visual data will be invaluable for the linguistic study of signed languages and the gestural component of spoken languages. The corpora will be used in training computer models, thereby leading to advances in both recognition and generation of signed languages and gesture. The tools will have educational applications for the teaching of ASL and other signed languages. Ultimately, this research will have implications for human-computer interfaces, enabling users to interact with computers via sign language or via a combination of speech and gesture, and may also lead to alternate input mechanisms for disabled users as well as techniques for automatic recognition and classification of human actions.