EAGER: VizWiz - Enabling Blind People to Answer Visual Questions On-the-Go with Remote Automatic and Human-Powered Services

University Of Rochester, Rochester NY

Investigators

Abstract

The lack of access to visual information like text labels, icons, and colors frustrates blind people and severely decreases their independence. Current access technology uses fully-automatic approaches to address some problems in this space but is error-prone, limited in scope, and expensive. Blind people who can afford to do so must carry multiple special-purpose portable devices with different audio and tactile interfaces in order to access critical data about their environment such as product information from bar codes and location information via GPS. These devices would likely be used more often if they had greater functionality and failed less often. Providing a fallback by making it easy to consult a human assistant could be part of the solution. The PI's talking VizWiz application for mobile phones is one such prototype that connects blind people to remote human workers who answer general questions about the users' visual environments. VizWiz currently allows blind users to take a picture, speak a question, and receive answers back quickly. Preliminary findings have demonstrated the potential advantages of including humans in the loop to help overcome visual problems that are still too difficult to be solved by automatic approaches alone, but questions remain about the efficacy, privacy, speed, and cost of these approaches. In this project the PI will seek answers to some of these questions, by conducting a longitudinal study of VizWiz with blind people to better understand how the application might fit into their everyday lives. He will endeavor to determine how users' existing social networks might be employed as a source of answers using applications for Facebook and Twitter. And he will seek to define new services with appropriate interfaces that let users mediate between automatic and human-powered remote sources for answers. A mobile accessibility solution using both automated and human Web services represents a significant advance in accessibility but presents challenging user interface questions. Understanding issues such as those enumerated above is necessary for human-powered services to be accepted as part of assistive technology. Broader Impacts: This exploratory research represents a new paradigm in human-computer interaction in which humans are both clients and providers. VizWiz has the potential to improve the independence of blind people, and may be both less expensive and more sustainable than current accessibility solutions. This project will improve our understanding of the types of tools that would be useful for blind people regardless of what is possible today with automatic computer vision, and will help us better understand how to recruit people to answer questions while respecting the asker's values. The research will involve blind people throughout; the resulting interfaces and functionality will be evaluated by blind people in the world going about their everyday lives. The interfaces, applications, and framework created and improved as part of this project will be released as open source so other researchers may build on the PI's results. Project outcomes will be broadly applicable to other problems where automated solutions may occasionally need human intervention.