CHS: Medium: Collaborative Research: Increasing Communication Rates through a TActile Phonemic Sleeve (TAPS)

Purdue University, West Lafayette IN

Investigators

Abstract

How much information can be transmitted through the skin? Conventional wisdom would say not as much as vision or hearing. The researchers argue, however, that there is a need to better understand how to "talk" to the skin in its own language. The team has recently devised a sleeve with 24 vibrators that delivers speech sounds in the form of distinct vibration patterns. Testing results from more than 100 normal-hearing individuals (in addition to one participant with a severe hearing loss) show that the vibration-based representation of speech sounds can be easily learned and combined to "sound out" English words in isolation. The best learners have demonstrated an ability to learn one English word per minute, up to a vocabulary size of 500 English words. This is an exciting breakthrough that can lead to a communication device for people with severe hearing impairments and for those whose hearing is temporarily compromised due to noisy environments such as firefighters. The main objective of this project is to systematically explore ways that the communication rate through the sleeve can be increased. Our goal is to reach at least 60-80 words per minute, a communication rate that has been demonstrated by the Tadoma method of speech communication, in which persons who are both deaf and blind place a hand over the face and neck of the talker to understand speech. Such a communication rate will make the sleeve a practical solution for the community of people who need an alternate means for speech communication, such as the ones with both hearing and vision impairments. If successful, the project will facilitate a better quality of life (in terms of social interactions as well as employability) for this disadvantaged community. This project addresses the fundamental question of the information capacity of the sense of touch within the context of increasing communication rates through a haptic display of speech. The project leverages the team's research and development experience on: (i) natural speech communication methods used by the deaf and deaf-blind; (ii) wearable haptic technologies; (iii) an information-theoretical approach to assessing human performance; and (iv) psychophysical properties of the tactual sense. The project builds upon the researchers' recent success at demonstrating their ability to develop tactile codes for the 39 phonemes of English that were highly recognizable in isolation and in words formed from sequences of phonemes. The research is focused on increasing the rate at which information can be transmitted through touch by trained participants that include observers with normal hearing as well as those with sensorineural hearing impairments who represent a major category of potential end users of the device. The overall goal is to achieve transmission rates of tactile speech stimuli that are comparable to those achieved by deaf-blind users of the Tadoma method, and to employ effective methods of training users of the display for reception of connected speech signals with a high degree of accuracy. Experiments will be carried out in which users of the tactile system are trained and tested on their ability to recognize phonemes, words, short phrases, and finally conversational sentences. Durational properties of the stimuli will be explored in these experiments with the aim of establishing the lower limits on phoneme duration and the intervals between phonemes and words. Estimates of effective communication rates will take into account both the transmission rate and accuracy of reception, with the goal of demonstrating effective communication rates on the order of 50-100 words per minute. A lower-bound estimate for haptic communication rate in terms of bits per second can then be derived from the transmission rate achieved at the end of this project. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.