CorpusLM: Towards a Unified Language Model on Corpus for Knowledge-Intensive Tasks

TL;DR

CorpusLM introduces a unified language model that integrates generative retrieval, closed-book generation, and retrieval-augmented generation to improve performance on knowledge-intensive tasks by effectively leveraging external corpora.

Contribution

The paper proposes CorpusLM, a novel unified model that combines generative retrieval and RAG with a new decoding process, enhancing knowledge-intensive task performance.

Findings

Outperforms existing models on KILT benchmark

Improves retrieval quality through ranking-oriented DocID generation

Enhances downstream task accuracy with unified decoding strategy

Abstract

Large language models (LLMs) have gained significant attention in various fields but prone to hallucination, especially in knowledge-intensive (KI) tasks. To address this, retrieval-augmented generation (RAG) has emerged as a popular solution to enhance factual accuracy. However, traditional retrieval modules often rely on large document index and disconnect with generative tasks. With the advent of generative retrieval (GR), language models can retrieve by directly generating document identifiers (DocIDs), offering superior performance in retrieval tasks. However, the potential relationship between GR and downstream tasks remains unexplored. In this paper, we propose \textbf{CorpusLM}, a unified language model that leverages external corpus to tackle various knowledge-intensive tasks by integrating generative retrieval, closed-book generation, and RAG through a unified greedy decoding process. We design the following mechanisms to facilitate effective retrieval and generation, and improve the end-to-end effectiveness of KI tasks: (1) We develop a ranking-oriented DocID list generation strategy, which refines GR by directly learning from a DocID ranking list, to improve retrieval quality. (2) We design a continuous DocIDs-References-Answer generation strategy, which facilitates effective and efficient RAG. (3) We employ well-designed unsupervised DocID understanding tasks, to comprehend DocID semantics and their relevance to downstream tasks. We evaluate our approach on the widely used KILT benchmark with two variants of backbone models, i.e., T5 and Llama2. Experimental results demonstrate the superior performance of our models in both retrieval and downstream tasks.