CRII: RI: Large-Scale Discovery and Organization of Subcategories and Parts from Image and Video Segments

Oregon State University, Corvallis OR

Investigators

Abstract

This project develops a system for understanding of visual categories given limited annotations. The system automatically detects subcategories and object parts with only category-level annotations. Such detailed understanding is important for autonomous systems to perform interactions with objects or to recognize them under occlusion. For this purpose, current deep neural networks will be extended to better support objects and parts of irregular shape. Once understandings at such level have been achieved, it helps to construct new categories from just a few exemplars, which has broad applications in autonomous systems. This research generalizes previously successful approaches in semantic segmentation and unsupervised video segmentation for an efficient approach to learn subcategories and parts. The framework starts from overlapping figure-ground segment proposals, computes least squares regressors from input segments against segment overlaps, and utilizes the Sherman-Morrison-Woodbury formula and structures from the quadratic loss function for efficient optimization of thousands to hundreds of thousands of subcategories and parts simultaneously. This project then explores the training of deep convolutional networks with initializations from these subcategories and parts defined on free-form segments. This requires generalization of the neural network architecture to handle free-form segments that can deform through a video sequence. It is proposed to use the geodesic distance transform on spatial-temporal segments to define customized filters for different localities for improved performance and better interpretability.