CAREER: Natural User Interfaces for Children

University Of Florida, Gainesville FL

Investigators

Abstract

Natural user interfaces allow users to interact with technology through modalities like touch, gesture, and motion. They are a key element in realizing the vision of ubiquitous computing, yet they present challenges with respect to supporting children. The PI's research on touchscreen interaction for children has found that existing surface gesture recognition algorithms designed, trained, and tested on adult input, and interaction design guidelines developed based on adult interaction patterns, do not apply equally well to children. For example, if a typical gesture interface is expecting a gesture to be entered as a single stroke, the system will not be able to process the multiple strokes generated by a child, leading to an unsuccessful interaction for that child. Recognition for whole-body interaction gestures experiences similar challenges; a child is more likely to perform an action or gesture (e.g., "jump" or "wave") with greater intensity or different motion paths than an adult performing the same gesture. In this research the PI's goal is to fundamentally advance our understanding of how to design and develop natural user interactions for children. The research will be carried out in three phases: (1) Data Collection and Analysis: collection of input behaviors from elementary-school aged children in each modality, and analysis for patterns and characteristics of children's input; (2) Recognition and Classification: development of new recognition algorithms attuned to the expected input behavior patterns of children and use of machine learning to evaluate their performance; and (3) Multimodal Interaction: investigation of multimodal input patterns exhibited by children, and validation of new approaches to multimodal synthesis that perform well on children's natural input. A testbed application will be developed to showcase the findings in an educational domain. Open-source natural user interaction recognition and synthesis algorithms and clear, practicable design recommendations will be developed and released in both peer-reviewed papers and on the project website for use by researchers and practitioners. This research will make fundamental advances in our understanding of child-computer interaction with natural user interfaces and develop robust new approaches for recognizing input and recovering from errors. This work will contribute solutions to interaction design research questions, such as how to best adapt and use these new modalities for children, and machine learning research questions, such as how to develop intelligent multimodal recognition algorithms tailored for children's input. The knowledge gained and contributions delivered by this research will inform the design of ubiquitous computing for children in learning contexts. To these ends, the PI will create models of expected input behavior patterns for children in two natural modalities, touchscreen interaction and whole-body interaction, and use these models to develop and adapt intelligent recognition algorithms tailored to process children's input. Multimodal interaction, or streamlined processing of disparate input from multiple simultaneous and unsynchronized input streams, is also a key component of natural user interfaces. Like unimodal recognition, traditional approaches to multimodal fusion for adult input may also not apply well to children. Specifically, the following research questions are targeted: What are the ways children produce interaction behaviors in multimodal natural user input modalities? What interaction design techniques are most effective for children using multimodal natural user input modalities? What new multimodal recognition and fusion algorithms for natural user interaction modalities perform effectively on children's input? This work will be conducted in partnership with schools and teachers, allowing the research findings to have immediate impact on how real children are using natural user interaction technology in educational contexts. The project will involve both undergraduate and graduate research assistants, broadening participation in computer science by recruiting women and underrepresented minority students. Education plans focus on developing a new undergraduate certificate in Human-Centered Computing.