CHS: Medium: Improving the Accessibility of Mobile Applications by Enabling Third-Party Assessment, Repair, and Enhancement

University Of Washington, Seattle WA

Investigators

Abstract

Many mobile applications (apps) do not properly implement accessibility services that help people with disabilities use computers, effectively denying access to apps and associated services for many vulnerable people. This project will develop new approaches to assess, repair, and enhance accessibility across large numbers of apps without needing access to source code of an app or the phone's operating system. To do this the research team will develop new pixel-level analyses of app interfaces to assess whether they correctly implement application programming interfaces (APIs) that the phone platform provides to support accessibility, then release datasets that give a picture of accessibility across the ecosystem of apps. The team will also develop new approaches that allow third parties to build alternate "proxy" interfaces that run on top of an existing app and implement accessibility APIs, as well as new approaches for those proxies to be personalized to a particular person's abilities and needs. The team will work closely with partners who serve people with disabilities in both design and evaluation of the tools, bringing direct benefits to those partners, as well as broader social benefits through actively disseminating the tools, methods, and datasets to support others in doing accessibility research. The project will also provide research and education opportunities for students from grade to grad school, with the team specifically targeting students from groups that are underrepresented in computing research, including those with disabilities. For assessment, the team will develop methods for executing an application in a mobile phone emulator, navigating its interface using common tasks, capturing both images of the interface and its accessibility API representation, and evaluating whether a recovered pixel-based interface hierarchy from the captured image correctly corresponds to the API representation. Working with people with disabilities to develop effective summaries and presentations of the analyses, the team will develop scalable infrastructure and efficient analyses to look for patterns in types of accessibility failures, how these failures are distributed across apps and developers, and how they are affected by app updates. For repair and enhancement, the team will develop interaction proxies that leverage the platform's accessibility services to create overlay windows that cover the underlying app interface and exchange events with it. They will also develop design patterns that link common accessibility failures with repair strategies, providing libraries to quickly implement these patterns in interaction proxies to support both third-party and eventually automatic repair of these common failures. Finally, the team will explore how novel interaction techniques that respond to individual abilities could be broadly deployed via interaction proxies, using a Model-View/Controller paradigm to manage the combinatorial complexity of applying particular enhancements to particular interfaces.