On the Merits of LLM-Based Corpus Enrichment

TL;DR

This paper explores using large language models to enrich document corpora, enhancing retrieval effectiveness and supporting applications like retrieval augmented generation and answer attribution in question answering.

Contribution

It introduces a novel approach of leveraging genAI for corpus enrichment to improve information retrieval and related tasks, supported by empirical experiments.

Findings

Enriched documents are more retrievable than original ones.

Corpus enrichment improves retrieval performance.

Potential benefits for retrieval augmented generation and answer attribution.

Abstract

Generative AI (genAI) technologies -- specifically, large language models (LLMs) -- and search have evolving relations. We argue for a novel perspective: using genAI to enrich a document corpus so as to improve query-based retrieval effectiveness. The enrichment is based on modifying existing documents or generating new ones. As an empirical proof of concept, we use LLMs to generate documents relevant to a topic which are more retrievable than existing ones. In addition, we demonstrate the potential merits of using corpus enrichment for retrieval augmented generation (RAG) and answer attribution in question answering.