LightRetriever: A LLM-based Text Retrieval Architecture with Extremely Faster Query Inference

TL;DR

LightRetriever introduces a lightweight LLM-based text retrieval system that drastically speeds up query inference by replacing full query encoding with simple embedding lookups, while maintaining high retrieval accuracy.

Contribution

It proposes a novel approach that retains full document encoding with a minimal query encoding workload, enabling extremely faster retrieval without sacrificing performance.

Findings

Over 1000x speedup in query encoding compared to full LLMs

More than 10x increase in end-to-end retrieval throughput

Maintains 95% of retrieval performance across benchmarks

Abstract

Large Language Models (LLMs)-based text retrieval retrieves documents relevant to search queries based on vector similarities. Documents are pre-encoded offline, while queries arrive in real-time, necessitating an efficient online query encoder. Although LLMs significantly enhance retrieval capabilities, serving deeply parameterized LLMs slows down query inference throughput and increases demands for online deployment resources. In this paper, we propose LightRetriever, a novel LLM-based retriever with extremely lightweight query encoders. Our method retains a full-sized LLM for document encoding, but reduces the workload of query encoding to no more than an embedding lookup. Compared to serving a full LLM on an A800 GPU, our method achieves over 1000x speedup in query encoding and over 10x increase in end-to-end retrieval throughput. Extensive experiments on large-scale retrieval benchmarks show that LightRetriever generalizes well across diverse tasks, maintaining an average of 95% retrieval performance.