EAGER: Joint Modeling and Querying of Social Media and Video Data

University Of California-Riverside, Riverside CA

Investigators

Abstract

As the amount of user generated data increases, it becomes more challenging to effectively search this data for useful information. There has been work on how to search text social media posts, such as Tweets, or videos. However, searching on these sources using separate tools is ineffective because the information links between them are lost; for instance, one cannot automatically match social network posts with activities seen on a video. As an example, consider a set of tweets and videos (which may be posted on Twitter or other media) generated during a riot. A police detective would like to jointly search this data to find material related to a specific incident like a car fire. Some tweets (with no contained video) may comment on the car fire, while a video segment from another tweet shows the car during or after the fire. Linking the videos with the relevant social media posts, which is the focus of this project, can greatly reduce the effort in searching for useful information. The successful completion of this project has the potential to improve the productivity of people who search in social media, such as police detectives, journalists of disaster management authorities. This project will also strengthen and extend the ongoing undergraduate research and high school outreach activities of the investigators. The objective of this project is to focus on the fundamental research tasks that would allow for joint modeling of social network and video data. Then, given a set of posts, the system would find relevant video segments and vice versa, by defining a common feature space for social media and video data. This proof-of-concept project will be evaluated on posts and videos shared on the Twitter platform. This is the right time to tackle this problem given the recent advances in deep learning and big data management technologies. A key risk is that the semantics in a tweet may not be enough to map it to a video segment; for that, the context (e.g., tweets from closely related users) of the tweet may need to be leveraged.