Collaborative Research: Quantifying sign reduction in sign language using human pose estimation

Trustees Of Boston University, Boston

Investigators

Abstract

The project investigates the phenomenon of lexically conditioned phonetic variation in sign language. This refers to how the pronunciation of a sign can vary based on the properties of signs, such as frequency of occurrence or the existence of similar-sounding signs (referred to as phonological neighbors) in the mental lexicon. In spoken languages, words that are used frequently may be phonetically reduced, that is, articulated with shorter, contracted vowel space as compared to low-frequency words. On the other hand, words with many phonological neighbors may be phonetically enhanced, e.g., produced with expanded vowel space compared to more isolated words. The articulators used to produce speech (the vocal tract) are fundamentally different from those used to produce signs (hands, face, body). This modality difference may lead to distinct predictions about phonetic variation. Limited research has been done on how these principles apply in signed languages, primarily due to a lack of large-scale machine-readable sign language datasets and adequate techniques to extract phonetic measurements from signed videos. The researchers leverage computer vision and the lexical database for a sign language that they developed under prior National Science Foundation (NSF) support, to analyze patterns of phonetic reduction in over 100,000 videos of signs produced by deaf signers. Specifically, they use human pose estimation techniques to extract signers’ anatomical landmarks and joint positions from the videos and transform the estimates into meaningful phonetic measurements, such as dispersion of the signers’ hands in the signing space. By examining the interactions between the underlying and surface properties of signs, this research provides insights into how the linguistic structure of sign languages affects sign articulation. By understanding the phonetic properties of signs, this research facilitates the development of sign recognition technologies, benefiting the signing community. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.