CHS: Small: RUI: Creation of Assistive Technology for the Blind Through Largescale Co-Design

Franklin W. Olin College Of Engineering, Needham MA

Investigators

Abstract

People who are blind face significant challenges, including a heightened risk of depression and a rate of participation in the labor force less than half that of the general population. Among other factors, difficulties with orientation and mobility (O&M) play a key role in the challenges faced by members of this community. Engineers have long tried to use technology to assist with O&M, but despite many attempts the impact of these efforts has been modest. This mixed track record is due to both shortcomings in technology (e.g., expensive yet unreliable hardware) as well as in methodology (e.g., not useful to a broad sample of users). While each of these shortcomings could be addressed in isolation, they can be most effectively addressed together. To this end, this project will create an infrastructure to enable two novel capabilities: designing O&M technology with people all around the world; and creating O&M technology with state-of-the-art flexibility and accuracy. These capabilities will be based on smartphones, which have recently been updated with 3D-sensing algorithms to support augmented reality; while these features were primarily designed to support experiences for sighted people, they have the potential to be repurposed as a robust platform for O&M technology. Project outcomes will provide enhanced support for independence and job-readiness for the eight million Americans who are blind. Additional broad impacts will derive from educational and outreach activities, including a summer research experience for 18 undergraduates at least 3 of whom will be blind, and a course on assistive technology and user-centered design where undergraduates will work directly with people in the community to improve accessibility. To achieve its objectives, the project will create a data-driven design process to allow a large, distributed team to work together to build O&M technology, ultimately recruiting dozens of co-designers (both blind and sighted) towards the goal of creating systems that are state-of-the-art in their robustness and usability. To recruit a large team of co-designers, the project will leverage the user base of the Clew smartphone app for O&M, which is used by 1,000 unique users per month. Smartphone-based features for O&M assistive technology will be evaluated, providing narrative feedback coupled with rich data streams utilizing a smartphone's sensors. The collected sensor data will be processed using novel machine learning and crowdsourcing algorithms to extract information (e.g., the user's location and the location of nearby objects), creating a first-of-its-kind dataset. The project will train machine learning algorithms on this dataset to create layers of functionality on top of the AR features of smartphones, to enable advanced capabilities such as the diagnosis and repair of motion-tracking errors. The project team will collaborate with people who are blind to incorporate these new capabilities into Clew, advancing the state-of-the-art in indoor navigation and exploration technology. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.