BIGDATA: F: DKM: Collaborative Research: Making Big Data Active: From Petabytes to Megafolks in Milliseconds

University Of California-Riverside, Riverside CA

Investigators

Abstract

A wealth of digital information is being generated daily through social networks, blogs, online communities, news sources, and mobile applications in an increasingly sensed world. Organizations and researchers recognize that tremendous value and insight can be gained by capturing this emerging data and making it available for querying and analysis. First-generation Big Data management efforts have been passive in nature -- queries, updates, and/or analysis tasks were mainly scaled to handle very large volumes of data. In contrast, this project will develop new techniques for continuously and reliably capturing Big Data collections (arising from social, mobile, Web, and sensed data sources) and will enable timely delivery of the right information to the relevant end users. In short, this project will provide a scalable foundation for moving from Big Passive Data to Big Active Data. Techniques should be developed to enable the accumulation and monitoring of petabytes of data of potential interest to millions of end users; when "interesting" new data appears, it should be delivered to end users in a time frame measured in (100's of) milliseconds. This project will build such an Active Big Data Management system and make it available as open source to the community. Students will be trained in technologies related to Big Active Data management and applications; such training is critical to addressing the information explosion that social media and the mobile Web are driving today. The general-purpose foundation for active information dissemination from Big Data will have broader impacts in areas such as public safety and public health. There are many challenges involved in building a foundation for Big Active Data. On the "data in" side, these include resource management in very large scale, LSM-based storage systems and the provision of a highly available, elastic facility for fast data ingestion. On the "data processing" side, challenges include the parallel evaluation of a large number of declarative data subscriptions over multiple) highly partitioned data sets. Amplifying this challenge is a need to efficiently support spatial, temporal, and similarity predicates in data subscriptions. Big Data also makes result ranking and diversification techniques critical in order for large result sets to be manageable. On the "data out" side, challenges include the reliable and timely dissemination of data of interest to a sometimes-connected subscriber base of unprecedented scale. As a software base, this project will be jump-started by using AsterixDB(http://asterixdb.ics.uci.edu/), an open-source Big Data Management System that supports the scalable storage, searching, and analysis of mass quantities of semi-structured data. For further information see the project web sites at https://www.ics.uci.edu/BigActiveData and http://www.cs.ucr.edu/~tsotras/BigActiveData