The Web Is Your Oyster - Knowledge-Intensive NLP against a Very Large Web Corpus

TL;DR

This paper explores knowledge-intensive NLP in an open web environment using a large web snapshot called Sphere, demonstrating that retrieval from Sphere can outperform Wikipedia-based models on several tasks despite challenges of scale and quality.

Contribution

It introduces a new evaluation setup using Sphere, a web-scale knowledge source, and shows how it can enhance NLP tasks beyond traditional Wikipedia-based approaches.

Findings

Sphere enables state-of-the-art performance on multiple tasks.

Dense indexing outperforms sparse BM25 on Wikipedia, but not on Sphere.

Shared infrastructure promotes further research in open-domain NLP.

Abstract

In order to address increasing demands of real-world applications, the research for knowledge-intensive NLP (KI-NLP) should advance by capturing the challenges of a truly open-domain environment: web-scale knowledge, lack of structure, inconsistent quality and noise. To this end, we propose a new setup for evaluating existing knowledge intensive tasks in which we generalize the background corpus to a universal web snapshot. We investigate a slate of NLP tasks which rely on knowledge - either factual or common sense, and ask systems to use a subset of CCNet - the Sphere corpus - as a knowledge source. In contrast to Wikipedia, otherwise a common background corpus in KI-NLP, Sphere is orders of magnitude larger and better reflects the full diversity of knowledge on the web. Despite potential gaps in coverage, challenges of scale, lack of structure and lower quality, we find that retrieval from Sphere enables a state of the art system to match and even outperform Wikipedia-based models on several tasks. We also observe that while a dense index can outperform a sparse BM25 baseline on Wikipedia, on Sphere this is not yet possible. To facilitate further research and minimise the community's reliance on proprietary, black-box search engines, we share our indices, evaluation metrics and infrastructure.