DS SERVE: A Framework for Efficient and Scalable Neural Retrieval

TL;DR

DS-Serve is a scalable neural retrieval framework capable of handling massive text datasets with low latency and flexible trade-offs, supporting various applications like RAG and data attribution.

Contribution

It introduces a novel framework that efficiently transforms large-scale datasets into high-performance neural retrieval systems with flexible inference options.

Findings

Supports half a trillion tokens dataset processing

Achieves low latency and modest memory usage on a single node

Enables flexible trade-offs between latency, accuracy, and diversity

Abstract

We present DS-Serve, a framework that transforms large-scale text datasets, comprising half a trillion tokens, into a high-performance neural retrieval system. DS-Serve offers both a web interface and API endpoints, achieving low latency with modest memory overhead on a single node. The framework also supports inference-time trade-offs between latency, accuracy, and result diversity. We anticipate that DS-Serve will be broadly useful for a range of applications, including large-scale retrieval-augmented generation (RAG), training data attribution, training search agents, and beyond.