Profiling Web Archive Coverage for Top-Level Domain and Content Language

TL;DR

This paper investigates optimizing web archive queries by targeting only the most relevant archives based on domain and language, reducing query load while maintaining comprehensive TimeMaps in most cases.

Contribution

It introduces a profiling method to identify key archives for specific domains and languages, enabling more efficient querying of web archives.

Findings

Querying top three archives yields 84% of complete TimeMaps.

Profiling reduces query volume by 75%.

Focus on relevant archives improves efficiency without significant loss.

Abstract

The Memento aggregator currently polls every known public web archive when serving a request for an archived web page, even though some web archives focus on only specific domains and ignore the others. Similar to query routing in distributed search, we investigate the impact on aggregated Memento TimeMaps (lists of when and where a web page was archived) by only sending queries to archives likely to hold the archived page. We profile twelve public web archives using data from a variety of sources (the web, archives' access logs, and full-text queries to archives) and discover that only sending queries to the top three web archives (i.e., a 75% reduction in the number of queries) for any request produces the full TimeMaps on 84% of the cases.