Large Language Models as Foundations for Next-Gen Dense Retrieval: A Comprehensive Empirical Assessment

TL;DR

This paper provides a comprehensive empirical assessment of large language models as backbone encoders for dense retrieval, highlighting their advantages in accuracy, generalization, and versatility across various retrieval tasks.

Contribution

It systematically evaluates over 15 models, revealing how size and pretraining influence retrieval performance and generalization capabilities.

Findings

Larger models improve in domain accuracy and data efficiency.

Extensive pretraining enhances retrieval performance.

Larger models excel in zero shot and multi-task retrieval scenarios.

Abstract

Pretrained language models like BERT and T5 serve as crucial backbone encoders for dense retrieval. However, these models often exhibit limited generalization capabilities and face challenges in improving in domain accuracy. Recent research has explored using large language models (LLMs) as retrievers, achieving SOTA performance across various tasks. Despite these advancements, the specific benefits of LLMs over traditional retrievers and the impact of different LLM configurations, such as parameter sizes, pretraining duration, and alignment processes on retrieval tasks remain unclear. In this work, we conduct a comprehensive empirical study on a wide range of retrieval tasks, including in domain accuracy, data efficiency, zero shot generalization, lengthy retrieval, instruction based retrieval, and multi task learning. We evaluate over 15 different backbone LLMs and non LLMs. Our findings reveal that larger models and extensive pretraining consistently enhance in domain accuracy and data efficiency. Additionally, larger models demonstrate significant potential in zero shot generalization, lengthy retrieval, instruction based retrieval, and multi task learning. These results underscore the advantages of LLMs as versatile and effective backbone encoders in dense retrieval, providing valuable insights for future research and development in this field.