CPS: Small: Collaborative Research: RF Sensing for Sign Language Driven Smart Environments

University Of Alabama Tuscaloosa, Tuscaloosa AL

Investigators

Abstract

Deaf individuals who rely on American Sign Language (ASL) as their primary mode of communication heavily rely on technology as an assistive device. Yet, many technologies are designed for hearing individuals, which precludes the Deaf community from benefiting from advances, which, if designed to be compatible with ASL, could in fact generate tangible improvements in their quality of life. This proposal aims at transforming ubiquitous sensing technologies through the integration of a new sensing modality - radio frequency (RF) sensing - into smart environments designed to respond to the needs of ASL users. RF sensors are uniquely desirable for this application because they are non-contact, can operate in the dark or through-the-wall, protect privacy, and bring to bear a new type of information that will aid in ASL understanding: namely, the micro-Doppler signature, which is reflective of the time-varying velocity profiles of motion. Thus, RF sensing is uniquely suited to capture the rapid progression of dynamic sign sequences that is characteristic of ASL usage. This collaborative project not only brings to bear, for the first time, a linguistic perspective to RF-based motion recognition, but also a physics-based machine learning approach achieved through integration of kinematics with deep learning. In this way, the project aims at 1) improving ASL recognition technologies and the design of smart environments for deaf individuals, 2) augmenting the tools linguists use to analyze language and related cognitive processes, and 3) advancing machine learning approaches specifically geared towards RF signal classification. The project is focused on developing signal processing algorithms for leveraging the unique aspects of RF sensing towards understanding of ASL and related linguistic features. More specifically, three aspects of ASL recognition are considered: classification of pre-defined ASL words and phrases, design of RF-sensing based dynamic sequence segmentation algorithms, and differentiation of daily activities from communicative sign language gestures. Novel ways of visualizing and representing RF data in one, two, and three dimensions will be investigated, both for extraction of linguistic features and as inputs to deep neural networks. Novel techniques will be developed for classification of three-dimensional time-varying data streams, the generation of synthetic RF data samples that have improved kinematic fidelity and realism, sequential classification and segmentation, as well as discrimination of daily motion from communicative signing. The critical experiments conducted during this project will result in a one-of-a-kind dataset of multi-frequency RF sensor network and Kinect(tm) sensor measurements of ASL signs, which will be made publicly available. The project directly engages the Deaf community through support and interaction of the Alabama Institute of Deaf and Blind (AIDB) and Gallaudet University as part of a needs-driven approach to communicative and assistive technology design, which will ultimately serve personal, professional, and educational needs of the Deaf community. This project is jointly funded by the Cyber Physical Systems Program and the Established Program to Stimulate Competitive Research (EPSCoR). This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.