SCH: Scene-Aware AR Systems for Low Vision People in Activities of Daily Living

University Of Wisconsin-Madison, Madison WI

Investigators

Abstract

Uncorrectable vision loss, a.k.a. low vision, can significantly impact activities of daily living (ADLs), affecting indepe ndence, health, and social well-being . Magnifiers are the most commonly used low vision aids, however, they largely reduce visual field. Augmented reality (AR) glasses present a unique opportunity f or low vision technology by intelligently recognizing the environment and enhancing users' vision based on their contexts. However, existing AR solut ions mainly rely on image processing techniques, generating pure pixel-level augmentations to the whole scene, such as pixel remapping for visual field loss, and recoloring based on distance. These solutions arbitrarily alter the user's full visual field with no semantic understanding of the scene, distorting users' natural vision and diminishing important visual details. As a result, they ca nnot effectively support complex ADLs that involve constant motion and object interactions, such as cooking in the kitchen, or safely navigating a crowded street. The overall objective of this proposal is to fundamentally advance low vision technology by leveraging state-of-the-art Al techniques and presenting unobtrusive scene-aware AR augmentations that are tailored to users' environments and visual abilities. Contextualized in two key ADLs (meal preparation, outdoor mobility), the research will investigate how to recognize and augment visual elements in AR from three fundamental dimensions via three specific aims: (A1) E nhancing perception of visual salience, (A2) E nhancing object affordance for safe and efficient interactions, (A3) E nhancing awareness of environmental dynamics. In each aim, the research will contribute to both HCI and Al by designing scene-aware AR augmentations with low vision users and rehabilitation professionals, and developing egocentric video datasets and Al models to enable fast and robust scene recognition in AR. In the final year, a holistic AR system will be developed and evaluated for real world impact and limitations via field studies (A4). With the Al-powered AR systems f or low vision, the project will contribute to the mission of N EI by fundamentally transforming conventional low vision aids and improving independence and quality of life for low vision people in various challenging ADLs. The involvement of low vision participants and rehabilitation professionals throughout the research will also increase their exposure to state-of-the-art technolog y, potentially increasing the technology acceptance and adoption in low vision rehabilitation. RELEVANCE (See instructions): The proposed research is relevant to public health because it investigates how to design and develop Al-powered AR systems to enhance low vision users' visual perceptions, empowering them to navigate activities of daily living with greater safety, efficiency, and independence. The project's contribution to low vision technology directly aligns with the mission of N EI , which emphasizes the importance of eliminating vision loss and improving quality of life through vision research.