III: Small: Ivory -- A Hadoop Toolkit for Distributed Text Retrieval

University Of Maryland, College Park, College Park MD

Investigators

Abstract

Text search is a technology that is vital for modern information-based societies. Today's systems face the daunting challenge of handling quantities of text previously unimaginable. Cluster computing is the only practical solution for addressing the issue of scale. This project leverages the MapReduce framework (via the open-source Hadoop implementation) to tackle issues of robustness and scalability in processing large amounts of data for information retrieval applications. More generally, the goals are to explore the relationship between processor, disk, memory, and network in large distributed computing environments, where many assumptions made in single machines no longer hold. One pertinent example is the fundamental mismatch between Hadoop and the demands of real-time interactive applications. Because it was designed for throughput-oriented batch processing, Hadoop currently does not provide low-latency disk access necessary for real-time search. A distributed in-memory object caching architecture provides a potential solution to this problem. To achieve broader impact, the results of this research will be implemented in Ivory, an open-source toolkit for distributed information retrieval built from the ground up with cluster architectures in mind. The availability of this toolkit will help sustain activities in the emerging area of "cloud computing". Additional information is available on the project website (http://www.umiacs.umd.edu/~jimmylin/cloud-computing).