CHS: Small: Improving Web Accessibility Through Multi-Resolution Mixed-Initiative Interaction Tools

Regents Of The University Of Michigan - Ann Arbor, Ann Arbor MI

Investigators

Abstract

This research will create powerful new ways to bridge critical accessibility gaps that people with visual impairments face on the Web every day, enabling greater independence and access to information. Many websites are difficult or impossible for users with visual impairments to access through screen readers. Many of these accessibility barriers have been shown currently to be insurmountable without help from a sighted colleague. This project envisions a future in which the Web can be made more navigable and accessible over time while using minimal human effort to bridge access gaps the first time they are encountered, and then automating this bridging support for future interactions. This is done through a mixed-initiative interaction between end users, web automation tools, and remote human workers, at multiple levels of interaction resolution. This allows development of a system that will not only work every time for end users (even on web pages the system has never seen before), but also to leverage the common design elements on the Web more effectively in order to reduce the total amount of human effort needed to successfully learn a new task, significantly reducing cost and improving scalability. In the approach to be developed by this project, when Web users with visual impairments encounter an inaccessible site, they will be able to specify their high-level intent in natural language and then a combination of automated macros and remote helpers will help them achieve their task. Not only do websites differ greatly, but websites often change their underlying structure, and users' goals change as well. Web automation scripts are often unable to infer how to achieve the end user's high-level intent under such dynamic conditions. Human crowd workers, on the other hand, are adept at figuring out which actions to take to achieve a given goal on a website. This work will address some of the fundamental challenges of using automation by employing humans to fill in the gaps where automation fails, and to understand when semantically-meaningful changes have occurred in how a page works. The research is divided into two phases: the first phase focuses on developing methods for creating macros, and the second phase will focus on ways to leverage crowd helpers to make these macros more robust. Both phases will advance the state of the art for Web automation. Progressively the system will be evaluated through a set of laboratory and field studies. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.