From Retrieval to Generation: Comparing Different Approaches

TL;DR

This paper systematically compares retrieval, generation, and hybrid models for knowledge-intensive tasks like ODQA, highlighting their strengths, weaknesses, and optimal use cases through extensive evaluation and analysis.

Contribution

It provides a comprehensive evaluation of different approaches, offering practical insights and detailed comparisons to guide future development in retrieval-augmented language modeling.

Findings

DPR achieves 50.17% top-1 accuracy on NQ in ODQA.

Hybrid models improve reranking scores on BEIR dataset.

Retrieval-based methods like BM25 have lower perplexity in language modeling.

Abstract

Knowledge-intensive tasks, particularly open-domain question answering (ODQA), document reranking, and retrieval-augmented language modeling, require a balance between retrieval accuracy and generative flexibility. Traditional retrieval models such as BM25 and Dense Passage Retrieval (DPR), efficiently retrieve from large corpora but often lack semantic depth. Generative models like GPT-4-o provide richer contextual understanding but face challenges in maintaining factual consistency. In this work, we conduct a systematic evaluation of retrieval-based, generation-based, and hybrid models, with a primary focus on their performance in ODQA and related retrieval-augmented tasks. Our results show that dense retrievers, particularly DPR, achieve strong performance in ODQA with a top-1 accuracy of 50.17\% on NQ, while hybrid models improve nDCG@10 scores on BEIR from 43.42 (BM25) to 52.59, demonstrating their strength in document reranking. Additionally, we analyze language modeling tasks using WikiText-103, showing that retrieval-based approaches like BM25 achieve lower perplexity compared to generative and hybrid methods, highlighting their utility in retrieval-augmented generation. By providing detailed comparisons and practical insights into the conditions where each approach excels, we aim to facilitate future optimizations in retrieval, reranking, and generative models for ODQA and related knowledge-intensive applications.