HCC-Small: End-User Retrofitting of Applications By Recognizing Text and UI Components

Brown University, Providence RI

Investigators

Abstract

Despite the enormous effort that goes into application design, end users invariably encounter workflow scenarios that are poorly supported by even the most developed, commercial-quality applications. Lack of support for end-user extension increases the burden on application developers to exhaustively anticipate the needs of all end users and then to commit enough resources to address their needs, which often includes re-implementing solutions already found in other applications. This project develops the feasibility of externally retrofitting, without altering, running applications with new interfaces and functionality made possible by using a combination of application-independent pixel-level recognition techniques and more specialized techniques for inspecting data structures exposed by the window system or application. Users will be able to make annotations to those running applications, including hand-drawn ink, typed text, diagrams, interactive widgets, or even links to other application user interface components. Registration techniques will then be researched to associate the annotations with specific elements of an application or document so that such an annotation can be made to appear perhaps only in one place in a specific file, or whenever a certain application runs. This supports a variety of practices, including: integrating functionality from different applications, enriched collaboration, task or user customized interfaces, and adding new fine-grained user interface elements to applications. Broader Impact. Making applications more malleable by embracing customization as a first class and general notion could fundamentally shift application design. Instead of relying solely on inherently slow development of one-size-fits-all applications, end-users could become de facto participants in application design, somewhat analogous to the relationship between traditional print-copy publishing and blogging. Rather than waiting, possibly years, for even simple revisions to fix workflow inefficiencies, users will be empowered to make limited modifications to applications almost as easily as marking up a document, for example by freely rearranging applications? interfaces, or adding diagrammatic annotations displayed over documents which do not natively support them.