Enabling Audio-Haptic Interaction with Physical Objects for the Visually Impaired

Smith-Kettlewell Eye Research Institute, San Francisco CA

Investigators

Linked publications & trials

Abstract

? DESCRIPTION (provided by applicant): Enabling Audio-Haptic Interaction with Physical Objects for the Visually Impaired Summary CamIO (Camera Input-Output) is a novel camera system designed to make physical objects (including documents, maps and 3D objects such as architectural models) fully accessible to people who are blind or visually impaired. It works by providing real-time audio-haptic feedback in response to the location on an object that the user is pointing to, which is visible to the camera mounted above the workspace. While exploring the object, the user can move it freely; a gesture such as double-tapping with a finger signals for the system to provide audio feedback about the location on the object currently pointed to by the finger (or an enhanced image of the selected location for users with low vision). Compared with other approaches to making objects accessible to people who are blind or visually impaired, CamIO has several advantages: (a) there is no need to modify or augment existing objects (e.g., with Braille labels or special touch-sensitive buttons), requiring only a low- cost camera and laptop computer; (b) CamIO is accessible even to those who are not fluent in Braille; and (c) it permits natural exploration of the object with all fingers (in contrast with approaches that rely on the use of a special stylus). Note also that existing approaches to making graphics on touch-sensitive tablets (such as the iPad) accessible can provide only limited haptic feedback - audio and vibration cues - which is severely impoverished compared with the haptic feedback obtained by exploring a physical object with the fingers. We propose to develop the necessary computer vision algorithms for CamIO to learn and recognize objects, estimate each object's pose to help determine where the fingers are pointing on the object, track the fingers, recognize gestures and perform OCR (optical character recognition) on text printed on the object surface. The system will be designed with special user interface features that enable blind or visually impaired users to freely explore an object of interest and interact with it naturally to access the information they want, which will be presented in a modality appropriate for their needs (e.g., text-to-speech or enhanced images of text on the laptop screen). Additional functions will allow sighted assistants (either in person or remote) to contribute object annotation information. Finally, people who are blind or visually impaired - the target users of the CamIO system - will be involved in all aspects of this proposed research, to maximize the impact of the research effort. At the conclusion of the grant we plan to release CamIO software as a free and open source (FOSS) project that anyone can download and use, and which can be freely built on or modified for use in 3rd-party software.