CAREER: Informal Tools for Multimodal User Interface Design

University Of California-Berkeley, Berkeley CA

Investigators

Abstract

This is the first year of funding of a 4 year continuing award. Multimodal user interfaces are likely to be the predominant interface of the future. These interfaces can take advantage of a wide variety of human perceptual and motor abilities, such as speech and gesture, to achieve a more natural human computer communication. This project addresses the current difficulty in building such interfaces. The P1 will create new interactive tools for designing multimodal user interfaces targeted at a range of devices, from the palmtop to the desktop to wall sized displays. These tools will support an iterative design methodology which supports the informal techniques designers currently use in the early stages of design, such as sketching, storyboarding, and "Wizard of Oz". Programming by demonstration will allow specification of events that are not screen based, such as speech I/O or body gesture input. The goal is to allow designers to rapidly create "rough cuts" of their ideas, which can be quickly tested and modified. When ready, the designer can use these tools to generate user interface code for a variety of devices. This code can then be extended with the final interface details and application code. As an example application, the PI will apply the new tools to develop a set of informal presentation tools for classroom lectures. He will carry out both informal and controlled experiments to evaluate the effectiveness of the design tools and to inforin iterative refinement of both the tools and the example application. This research will result in new techniques for specifying visual and nonvisual input, which will be applicable to both multimodal and unimodal interface design tools, so that people can more easily create interfaces than is currently the case. The findings will also impact systems for people with disabilities (especially the vision impaired), as well as for mainstream users working in non traditional settings such as mobile and wearable computing.