Collaborative Research: WebScales-Towards a Large Scale Metasearch Engine

University Of Illinois At Chicago, Chicago IL

Investigators

Abstract

This project is to develop enabling techniques for a large-scale metasearch engine that aims at covering a much larger portion of the Web and at the same time retrieving more up-to-date and more useful documents than existing search engines and metasearch engines. A metasearch engine is a system that supports unified access to multiple existing search engines. It is estimated that there are hundreds of thousands of search engines on the Web (including deep Web). The proposed metasearch engine (WebScales) will attempt to connect to as many of these search engines as possible. The focus of the research is on advancing highly scalable distributed information retrieval technology and the research issues to be investigated in this project include (1) how to discover these search engines automatically, (2) how to incorporate them into the proposed metasearch engine automatically, (3) how to identify potentially useful search engines to use for any given user query submitted to the metasearch engine, and (4) how to achieve high retrieval effectiveness.