The Impact of Main Content Extraction on Near-Duplicate Detection

TL;DR

This paper examines how different main content extraction methods affect near-duplicate detection in web search, highlighting the trade-offs between precision and recall in the context of an open search infrastructure.

Contribution

It compares five content extraction techniques and analyzes their impact on near-duplicate detection performance using ClueWeb data.

Findings

Full content extraction improves precision in near-duplicate detection.

Main content extraction enhances recall but may reduce precision.

Different extraction methods influence the similarity thresholds used in detection.

Abstract

Commercial web search engines employ near-duplicate detection to ensure that users see each relevant result only once, albeit the underlying web crawls typically include (near-)duplicates of many web pages. We revisit the risks and potential of near-duplicates with an information retrieval focus, motivating that current efforts toward an open and independent European web search infrastructure should maintain metadata on duplicate and near-duplicate documents in its index. Near-duplicate detection implemented in an open web search infrastructure should provide a suitable similarity threshold, a difficult choice since identical pages may substantially differ in parts of a page that are irrelevant to searchers (templates, advertisements, etc.). We study this problem by comparing the similarity of pages for five (main) content extraction methods in two studies on the ClueWeb crawls. We find that the full content of pages serves precision-oriented near-duplicate-detection, while main content extraction is more recall-oriented.