ScalingNote: Scaling up Retrievers with Large Language Models for Real-World Dense Retrieval

TL;DR

ScalingNote introduces a two-stage approach that leverages large language models to improve dense retrieval performance while controlling online latency, validated through theoretical and empirical results in industrial settings.

Contribution

The paper presents a novel two-stage method for scaling dense retrieval with LLMs, combining training and distillation to balance performance and online efficiency.

Findings

Outperforms end-to-end models in relevance and efficiency

Verifies the scaling law of dense retrieval with LLMs in industry

Enables cost-effective, scalable dense retrieval systems

Abstract

Dense retrieval in most industries employs dual-tower architectures to retrieve query-relevant documents. Due to online deployment requirements, existing real-world dense retrieval systems mainly enhance performance by designing negative sampling strategies, overlooking the advantages of scaling up. Recently, Large Language Models (LLMs) have exhibited superior performance that can be leveraged for scaling up dense retrieval. However, scaling up retrieval models significantly increases online query latency. To address this challenge, we propose ScalingNote, a two-stage method to exploit the scaling potential of LLMs for retrieval while maintaining online query latency. The first stage is training dual towers, both initialized from the same LLM, to unlock the potential of LLMs for dense retrieval. Then, we distill only the query tower using mean squared error loss and cosine similarity to reduce online costs. Through theoretical analysis and comprehensive offline and online experiments, we show the effectiveness and efficiency of ScalingNote. Our two-stage scaling method outperforms end-to-end models and verifies the scaling law of dense retrieval with LLMs in industrial scenarios, enabling cost-effective scaling of dense retrieval systems. Our online method incorporating ScalingNote significantly enhances the relevance between retrieved documents and queries.