CAREER: Towards sensing and understanding fine-grained body postures in daily life using intelligent wearables with acoustic sensing

Cornell University, Ithaca NY

Investigators

Abstract

Despite the one billion wearables in use every day, computers' ability to recognize human activities in daily settings is still limited. One key roadblock is the inability of wearables to sense behavior-relevant information such as limb posture (for both hands and feet) or fine-grained details such as facial expressions. This project will enable the next generation of wearables to continuously track and interpret a key set of fine-grained body postures (e.g., face, hands, limbs, eyes, tongue) in daily life using low-power, low-cost, and privacy-sensitive intelligent acoustic sensing technologies. The project outputs will help researchers and developers monitor and exploit a range of high-resolution data in everyday settings to significantly improve the performance of downstream applications in areas with positive societal impact, including accessibility, telemedicine, and activity recognition. As a demonstration, this project will use the new wearables to immediately improve the accessibility of computers for deaf and hard-of-hearing individuals as well as people with speech impairments, by advancing American Sign Language (ASL) and Silent Speech recognition. To achieve the desired goals, the research will employ an iterative and user-centered design process. First, a list of AI-powered wearables that use acoustic sensing to continuously track fine-grained postures will be developed. These wearables will be evaluated extensively and iteratively in both lab and real-world settings to ensure optimal user experience and performance, and to identify any remaining challenges. Next, the research will address the critical challenges of deploying these data-driven acoustic sensing technologies in everyday settings by using customized signal processing and AI algorithms (such as data simulation and synthesis, data augmentation, and edge computing) to improve the system's generalizability across users and its robustness in the presence of noise, and to minimize training efforts while protecting user privacy. Thirdly, the research will demonstrate how these new wearables can enhance computers' ability to understand complex human behaviors, which will naturally support users in two high-impact downstream applications: ASL and silent speech recognition. Throughout the design, development, and evaluation process, the work will be carried out in collaboration with experts in related fields (including wearable computing, AI, linguistics, and otorhinolaryngology), and with partners in the target community (including persons who are deaf and hard-of-hearing, and those with speech impairments). This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.