Automatically Generating Consistent User Interfaces for Multiple Appliances

Carnegie Mellon University, Pittsburgh PA

Investigators

Abstract

During the last four years, the PI and his students have been investigating the use of handheld devices to control all kinds of home, office and factory equipment (such as stereos, VCRs, telephones, copiers, FAX machines, and clocks), as well as the non-driving functions of automobiles (such as the air conditioning and navigation systems). They have developed a high-level XML-based language to specify an appliance's features from which a high quality user interface can be generated on a PocketPC personal digital assistant (PDA) or on a mobile phone. However, two important problems remain: to automatically generate consistent interfaces for the user across different appliances; and to automatically generate a combined user interface for multiple appliances that operate as a logical unit. No existing automatic generation system for user interfaces has addressed these problems. Solving the consistency problem would, for example, mean that people could set the time on the VCR and the time to start recording in the same way that they set their alarm clock. By providing the interface on the user's handheld, the same consistent way to set the time would be used in every place that time-setting is required on all appliances. Solving the combined interface problem would, for example, mean that for an entire entertainment system, the user would just need to press "play DVD" and the system would automatically turn on the TV, switch the TV to "input3," turn on the stereo, switch the stereo to "aux" input, turn off the cable box, turn on the DVD player, and finally cause the DVD to start playing. These are the scenarios the PI plans to tackle in this project. He will develop a system that uses the interdependencies among all of a user's connected appliances to automatically create a combined user interface. The intellectual merit of the proposed research lies in determining how to generate interfaces that are consistent with one another and how to combine interfaces for multiple appliances. For consistency, this includes the sub-problems of matching which parts of different appliances should be made consistent with each other, and then how consistency can be provided when the appliances may have more or fewer features for related functions. The PI will have to develop fundamental new knowledge about what aspects of consistency are most important to preserve across appliances, and how to embody that knowledge into rules that the handheld can use to automatically generate consistent user interfaces. For combining multiple appliances, the intellectual merit will include techniques for describing the interconnection among appliances and for combining pieces of multiple appliances' user interfaces together. Extensive user studies will inform the designs and verify the results. Broader Impacts: No one has previously tried to automatically generate consistent interfaces from specifications for different appliances or computer applications that share some or many functions. The results will be useful for all researchers and developers who are concerned with consistency across multiple user interfaces for appliances or computers. Similarly, no one has automatically created a combined user interface using models defined separately for separate appliances. The results of this research will make complex appliances much easier to use for the general public, so that they are able to make better use of their devices and also to transfer their knowledge to the operation of new, potentially more complicated, appliances.