SGER: Initial User Experiments Towards a Spoken Language Interface for Programming

University Of Utah, Salt Lake City UT

Investigators

Abstract

The PIs are exploring issues relating to the development of viable systems that would allow programmers to create Java programs via spoken English input. They have to this end designed and implemented a proof-of-concept prototype called NaturalJava which supports user input of simple Java programs via written English sentences. Their ability to improve the prototype is limited at present both by software architecture shortcomings and by a lack of understanding of how real users would prefer to interact with a spoken programming interface. What kinds of written and spoken English sentences would they use? What type of navigation/editing model would be effective in such an environment? To answer questions such as these, the PIs plan to conduct a Wizard-of-Oz user study in which they will invite Java programmers to write programs under controlled conditions. Subjects will write programs using a system that behaves like an improved version of NaturalJava; half of them will use a written English interface, and the other half will use a spoken English interface. A hidden expert Java programmer will play the role of NaturalJava, reading (or listening to) the commands and creating the source code. The PIs expect to learn a great deal by recording and transcribing the sessions, interviewing the programmers, and examining the resulting Java programs. At the end of the year the PIs expect to have acquired a solid understanding of the issues, so that they will then be in a position to detail and pursue their long-term agenda which, if successful, would prove especially valuable to people with certain kinds of disabilities. The research team will include a graduate student who is blind, and for whom most of the NSF funds are allocated; the PIs are not charging for their time.