CISE-MSI: DP: HCC: Training a Virtual Guide Dog for Visually Impaired People to Learn Safe Routes Using Crowdsourcing Multimodal Data

Cuny Borough Of Manhattan Community College, New York NY

Investigators

Abstract

This award is funded in whole or in part under the American Rescue Plan Act of 2021 (Public Law 117-2). Twelve million blind and visually impaired (BVI) people in US face challenges traveling outdoor independently, particularly in a complex urban environment. A guide dog can offer safe and comfortable travel experiences; however, only 5% of BVI people choose dogs mainly due to the cost. Existing mobile navigation apps utilizing GPS and digital maps to provide navigation services, but usually the maps do not offer enough information, such as sidewalk accessibility and dynamic information, to ensure a safe travel experience. To this end, this research studies the challenges and behaviors of BVI people when they travel outdoor independently, using machine learning approaches on crowdsourcing multimodal travel data. The outcome of the research includes a deep understanding of BVI people’s travel behaviors and two assistive mobile apps based on mixed and augmented reality technology: one for personalized trip planning and orientation and mobility training, and another for personalized navigation and travel assistance. The research has broader impact in improving the quality of life of BVI people, and helping local government agencies to better maintain sidewalks. From the educational perspective, the research provides unique training opportunities for students at the City University of New York, including underrepresented populations in STEM at various levels, from undergraduate (both 2-year and 4-year), to master and doctoral students. The project aims to answer the following research questions: 1) How to efficiently survey comprehensive sidewalk data in a complex urban environment? 2) How to identify all travel challenges of BVI users on sidewalks and learn their travel behaviors? 3) How to develop an electronic guide Dog mobile app to provide personalized travel guidance for BVI users with or even without a map service? The project develops a unique user-centric crowdsourcing approach to collect multimodal travel data by BVI people themselves and without manual labeling, and develops multimodal machine learning algorithms to learn the models of semantic sidewalks and travel behaviors of BVI people. The research also develops two novel accessible mobile apps to validate the effectiveness of the above models: a mixed reality-based app for trip planning and realistic orientation & mobility training simulation, for helping BVI people to build a mental map; and an augmented reality-based app using the semantic sidewalk map and the travel behavior models, for safe and comfortable travel assistance. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.