Exploiting Locality in Searching the Web
Joel Young, Thomas L. Dean

TL;DR
This paper reexamines claims that using locality can significantly improve web search efficiency by conducting experiments with a standardized framework and dataset, providing insights into the validity of previous results.
Contribution
It introduces a reproducible experimental framework and tools to evaluate locality-based web search strategies using the WT10g corpus, clarifying prior claims.
Findings
Locality-based search can improve efficiency under certain conditions.
Experimental results support some earlier claims but also highlight limitations.
The framework enables consistent replication and further research.
Abstract
Published experiments on spidering the Web suggest that, given training data in the form of a (relatively small) subgraph of the Web containing a subset of a selected class of target pages, it is possible to conduct a directed search and find additional target pages significantly faster (with fewer page retrievals) than by performing a blind or uninformed random or systematic search, e.g., breadth-first search. If true, this claim motivates a number of practical applications. Unfortunately, these experiments were carried out in specialized domains or under conditions that are difficult to replicate. We present and apply an experimental framework designed to reexamine and resolve the basic claims of the earlier work, so that the supporting experiments can be replicated and built upon. We provide high-performance tools for building experimental spiders, make use of the ground truth and…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsWeb Data Mining and Analysis · Advanced Image and Video Retrieval Techniques · Text and Document Classification Technologies
