CRI II-NEW: IIS: Omniview Multi-modal Sensor Laboratory for Understanding Human Interactions in Ubiquitous Environments

Clarkson University, Potsdam NY

Investigators

Abstract

Understanding how people interact with objects and with each other is an important research area in human-computer interaction, particularly in contexts where phones, cameras, sensors, voice assistants, robots, and other computing devices help people accomplish their goals. To study these interactions, this project will equip a lab with state of the art equipment for capturing human activity that doesn't require attaching markers, wires, or sensors to people, and develop software to manage the large amounts of captured data and interfaces that help researchers and designers make use of the data. Much of the envisioned fundamental research will focus on how age, gender, physical ability, and experience level affect the way people interact with objects, as well as on using social cues such as emotions, inter-person distances, and gestures to understand how people interact with each other. A number of researchers at the lead investigators' institution will use the lab for related projects around people- and object-aware technologies, including assistive robotics, self-driving vehicles, teams and collaboration, and smart environments. The lab will also provide training for a postdoctoral researcher and research opportunities for undergraduates, support a number of courses taught at the PIs' institution, and provide new opportunities for interdisciplinary research. The PIs will develop a temporally synchronized and spatially calibrated sensor system that includes force plates, microphones, RGB and infrared cameras, and Kinect sensors, and deploy it in 20x20x12 foot lab. Data will be synchronized using linear time codes (LTCs), including custom hardware previously developed by the PIs to add LTC data to Kinects, and managed by a large cluster of computers and network attached storage devices. The infrastructure will use computer vision and sound reconstruction algorithms on the raw sensor data to reconstruct 3D spatiotemporal data that provides a full range 3D visualization of human interactions. By combining data from multiple sensor modalities in 3D, the infrastructure enables research in strengthening recognition of gestures and emotions of people engaged in interactions by analysis of prosody changes, face points, body skeletons, spatiotemporal motions, linguistics, heat signatures, and emotion- or task-driven physical impact. Further, the PIs will develop large repositories of omniview multi-modal 3D spatiotemporal data, and conduct research on tying these repositories to sensors on next-generation ubiquitous devices to make them people- and object-aware.