RankMamba: Benchmarking Mamba's Document Ranking Performance in the Era of Transformers

TL;DR

This paper evaluates Mamba, a model based on state space models, for document ranking tasks, comparing its performance and efficiency to transformer models, and provides insights into its applicability in information retrieval.

Contribution

The study benchmarks Mamba's effectiveness in document ranking, highlighting its competitive performance and efficiency challenges relative to transformer models.

Findings

Mamba models perform comparably to transformer models with similar training setups.

Mamba has lower training throughput than efficient transformer implementations like Flash Attention.

The work provides a foundation for exploring Mamba in other IR tasks.

Abstract

Transformer structure has achieved great success in multiple applied machine learning communities, such as natural language processing (NLP), computer vision (CV) and information retrieval (IR). Transformer architecture's core mechanism\, -- \,attention requires $O(n^2)$ time complexity in training and $O(n)$ time complexity in inference. Many works have been proposed to improve the attention mechanism's scalability, such as Flash Attention and Multi-query Attention. A different line of work aims to design new mechanisms to replace attention. Recently, a notable model structure Mamba, which is based on state space models, has achieved transformer-equivalent performance in multiple sequence modeling tasks. In this work, we examine Mamba's efficacy through the lens of a classical IR task\, -- \,document ranking. A reranker model takes a query and a document as input, and predicts a scalar relevance score. This task demands the language model's ability to comprehend lengthy contextual inputs and to capture the interaction between query and document tokens. We find that \textbf{(1) Mamba models achieve competitive performance compared to transformer-based models with the same training recipe; (2) but also have a lower training throughput in comparison to efficient transformer implementations such as flash attention.} We hope this study can serve as a starting point to explore \mamba models in other classical IR tasks. Our \href{https://github.com/zhichaoxu-shufe/RankMamba}{code implementation} is made public to facilitate reproducibility. Refer to~\cite{xu-etal-2025-state} for more comprehensive experiments and results, including passage ranking.