EAGER: Learning Transferable Visual Features

Cuny City College, New York NY

Investigators

Abstract

Artificial intelligence and machine learning have shown great promise for many applications in computer vision, multimedia, robotics, autonomous driving, medical imaging analysis, assistive technology, etc. In order to obtain better performance, large-scale labeled data are generally required to train deep neural networks. To avoid extensive cost of collecting and annotating large-scale data, a major goal of machine learning is to exploit new algorithms to learn general features from limited labeled or unlabeled data. This project aims to explore self-supervised methods to learn general visual features across different modalities from large scale data without using any human-labeled annotations. The learned general visual features can then be transferred to many different applications, such as human activity analysis, 3D scene understanding, and assistive technologies. The research is tightly integrated with graduate/undergraduate education in the City University of New York, a minority serving institution and one of the most diverse campuses in the United States. Most prior work of visual feature learning has focused on a single modality of data. This research is to explore methods of learning transferable visual features from multiple modalities including texts, audios, images, videos, and 3D data as well as investigate new loss functions to find optimal features. In particular, the will conduct the following research tasks: (1) exploration of new algorithms to effectively learn transferable visual features across multimodalities without requesting human annotations of large scale data; (2) investigation of effective algorithms and loss functions for bridging the gap among different modalities to handle the different feature distributions from different modalities; and (3) evaluation and generalization of the proposed technologies on different applications including human activity analysis, 3D scene understanding, and medical image processing. The project will result in new algorithms to effectively learn transferable features from multimodality data including texts, images, videos, and 3D data without depending on data annotations. The work will lead to advances in computer vision and machine learning technologies and the outcome algorithms will be general and broadly applicable across different real-world applications. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.