ITR: Data on the Deep Web: Queries, Trawls, Policies and Countermeasures

University Of California-Berkeley, Berkeley CA

Investigators

Abstract

The volume of hypertext on the World Wide Web is dwarfed by the amount of data made available in networked databases, which has been estimated to be 400 to 550 times larger than the WWW hypertext. This proposal refers to this data as the Federated Facts and Figures on the Internet, or simply the FFF. The goal of this work is to explore the mechanisms for -- and consequences of -- aggressively leveraging this resource. The proposal has three aspects. First, it describes algorithms and systems for exploiting facts and figures on the Internet. In particular, it proposes adaptive query processing for the Telegraph system, to adjust to the volatility characteristic of the Internet. It also proposes extending Telegraph to ``trawl'' large amounts of data from the FFF, by running recursive queries over multiple data sources. The second goal of the proposal is to explore the ramifications of providing FFF tools to the broad Internet user base. This includes an investigation of policy -- both social and technical -- and the economic incentives and ramifications surrounding such policies. The third goal is to explore the design space of countermeasures that can prevent FFF technologies from being misused. .