Mixture-of-PageRanks: Replacing Long-Context with Real-Time, Sparse GraphRAG

TL;DR

This paper introduces MixPR, a sparse, PageRank-based retrieval algorithm that efficiently handles long-context tasks, outperforming existing methods while significantly reducing compute costs, and enabling real-time processing on CPU.

Contribution

The paper presents MixPR, a novel, sparse, PageRank-based retrieval method that improves efficiency and performance for long-context tasks compared to prior RAG approaches.

Findings

MixPR achieves state-of-the-art results on long-context benchmarks.

MixPR is highly compute-efficient, capable of embedding millions of tokens in seconds.

MixPR runs entirely on CPU, enabling practical real-time applications.

Abstract

Recent advances have extended the context window of frontier LLMs dramatically, from a few thousand tokens up to millions, enabling entire books and codebases to fit into context. However, the compute costs of inferencing long-context LLMs are massive and often prohibitive in practice. RAG offers an efficient and effective alternative: retrieve and process only the subset of the context most important for the current task. Although promising, recent work applying RAG to long-context tasks has two core limitations: 1) there has been little focus on making the RAG pipeline compute efficient, and 2) such works only test on simple QA tasks, and their performance on more challenging tasks is unclear. To address this, we develop an algorithm based on PageRank, a graph-based retrieval algorithm, which we call mixture-of-PageRanks (MixPR). MixPR uses a mixture of PageRank-based graph-retrieval algorithms implemented using sparse matrices for efficent, cheap retrieval that can deal with a variety of complex tasks. Our MixPR retriever achieves state-of-the-art results across a wide range of long-context benchmark tasks, outperforming both existing RAG methods, specialized retrieval architectures, and long-context LLMs despite being far more compute efficient. Due to using sparse embeddings, our retriever is extremely compute efficient, capable of embedding and retrieving millions of tokens within a few seconds and runs entirely on CPU.