CIF: AF: Small: Foundations of Multimodal Information Integration

Duke University, Durham NC

Investigators

Abstract

Data comes in all forms, including visual, text, medical records, and comments on social medial. This diverse (multimodal) data is critical when data is scarce, noisy, and uncertain. Different modalities can improve joint inference and decision making and allow producing (at extremely low-cost) results that were possible only with high-end devices and techniques before. In addition, inferring a condition from unexpected data sources is of paramount importance in disciplines ranging from marketing to health-care and defense. This project addresses these fundamental challenges with new mathematical and computational tools. Through new collaborations, the project has access to unique data and problems of significant impact in human well-being. A related online class also continues to grow and develop, with over 120,000 students so far. A unique summer immersion program will also involve undergraduate students in multimodal data science research. The exploitation of multimodal data is one of the unifying themes of this project. A further unifying theme is the underlying mathematical foundation: subspace modeling and embedding. Tools from subspace modeling in the form of learning multimodal low-rank representations, modeling multimodal sparse networks, and solving for big data matrix decompositions are here developed. A third unifying motif of this work is the ubiquitous consideration of computational efficiency. All the above is developed in three major components: classification and recognition, data augmentation, and network analysis. The project addresses critical problems such as multimodal face recognition, dynamic multimodal graph inference, gaze analysis, multimodal network analysis, and non-negative matrix factorization. The overall goal is to efficiently exploit and integrate multimodal data to help in joint inference and decision making.