CAREER: Coping with the Data Deluge - Algorithms, Human Aspects and Applications

California Institute Of Technology, Pasadena CA

Investigators

Abstract

Massive volumes of data are now generated in social, scientific, business and military applications. However, access to the data is limited, due to computational, bandwidth, power limitations, as well as human constraints such as attention and privacy. A fundamental problem is thus to obtain most useful information at minimum cost. Most existing techniques are heuristics without guarantees, do not scale or cannot cope with dynamic, distributed phenomena that change over time. In addition, existing algorithms typically disregard human aspects of the information gathering task. This research (1) develops principled algorithms for optimized information gathering in distributed, uncertain, dynamic domains, (2) studies human aspects of optimized information gathering, considering attention, perception and privacy as fundamental constraints and (3) pursues novel, real-world applications. To accomplish these goals, fundamentally new techniques in combinatorial optimization, probabilistic reasoning and decision theory are developed. The approaches are unified in the mathematical framework of submodular function optimization. In addition to developing theoretically well-founded, rigorous approaches, an important part of the research is evaluation on interdisciplinary, real world applications in social computing, scientific data analysis and stochastic optimization in sustainability. These applications of optimized information gathering have the potential to enable new classes of systems and services for science and society. Impact is achieved through collaboration with partners from industry (Microsoft Corporation) and governmental facilities (USGS and JPL), as well as by broad dissemination of the results through an integrated education plan, tutorials, shared data and open-source software, and interdisciplinary collaborations.