CHS: Small: High Resolution Motion Capture

University Of Utah, Salt Lake City UT

Investigators

Abstract

This project studies a new type of full-body motion capture technology which has been enabled by recent advances in deep learning and high-resolution digital cameras. Unlike classical motion capture systems which rely on suits with small attached spheres that serve as markers, this project introduces a new type of suit using a special printed pattern instead of any attachments. This pattern will contain a new type of markers with two distinct advantages: (1) the ability to automatically detect which marker is which, and (2) a significantly more dense set of markers than previous systems. The proposed approach is fully passive and therefore easy to use, relying only (a) indoor lighting or natural daylight and (b) a suit made of elastic fabric with a special printed pattern, but without any wires or batteries. The only required electronics are standard digital cameras, ranging from just a few cameras to massive multi-camera systems. This flexibility will allow us to support applications on various scales, from individual researchers or makers to large institutions or production studios. New technology for full-body capture, featuring higher accuracy while being easy to use, has the potential to impact research and clinical studies of human motion, e.g., in orthopedics, sports medicine, rehabilitation, physical therapy and ergonomics. High-quality human motion data can also facilitate better virtual or augmented reality systems and applications. By combining computer science and human motion, motion capture systems enables unique educational and outreach opportunities involving activities popular among young people, such as sports and gymnastics. The idea of using a new type of motion capture suit with texture-based markers recognized using artificial neural networks has not been explored before and opens up many interesting research questions such as "Which types of markers and suit textures will lead to the best detection and labeling results, despite large elastic distortions induced by the motion of the skin?" The proposed computing methodology requires training and validation of neural networks and contributes to research on the following topics: (1) synthetic data generation, (2) automated data augmentation, (3) confidence calibration of neural networks, in particular teaching neural networks to quantify the risk of errors in their own predictions. To further improve robustness and accuracy, the probabilistic output of neural networks will be combined with priors, such as a 3D deformable shape model; this can be done in a principled way via Bayesian inference, which leads to research problems combining continuous and discrete optimization. Finally, the high-resolution data produced by the envisioned system motivates research on improving the anatomical realism of deformable shape models, in particular data-driven modeling of joint kinematics and muscle activations. The envisioned full-body capture system will be designed to be easy to replicate and deploy at various institutions, clinics or studios. This project will share research results through common open source codebase, facilitating collaboration and data sharing. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.