FastMamba: A High-Speed and Efficient Mamba Accelerator on FPGA with Accurate Quantization

TL;DR

FastMamba is an FPGA-based accelerator that significantly improves the deployment efficiency of Mamba2 state space models by employing hardware-algorithm co-design, accurate quantization, and optimized nonlinear function approximation.

Contribution

This paper introduces FastMamba, a novel FPGA accelerator with hardware-algorithm co-design that enables efficient, accurate quantization and processing of Mamba2 models on resource-constrained devices.

Findings

Achieves 68.80× speedup over CPU and 8.90× over GPU on specific tasks.

Attains 6× higher energy efficiency than GPU for large models.

Successfully implements 8-bit quantization and nonlinear function approximation.

Abstract

State Space Models (SSMs), like recent Mamba2, have achieved remarkable performance and received extensive attention. However, deploying Mamba2 on resource-constrained edge devices encounters many problems: severe outliers within the linear layer challenging the quantization, diverse and irregular element-wise tensor operations, and hardware-unfriendly nonlinear functions in the SSM block. To address these issues, this paper presents FastMamba, a dedicated accelerator on FPGA with hardware-algorithm co-design to promote the deployment efficiency of Mamba2. Specifically, we successfully achieve 8-bit quantization for linear layers through Hadamard transformation to eliminate outliers. Moreover, a hardware-friendly and fine-grained power-of-two quantization framework is presented for the SSM block and convolution layer, and a first-order linear approximation is developed to optimize the nonlinear functions. Based on the accurate algorithm quantization, we propose an accelerator that integrates parallel vector processing units, pipelined execution dataflow, and an efficient SSM Nonlinear Approximation Unit, which enhances computational efficiency and reduces hardware complexity. Finally, we evaluate FastMamba on Xilinx VC709 FPGA. For the input prefill task on Mamba2-130M, FastMamba achieves 68.80\times and 8.90\times speedup over Intel Xeon 4210R CPU and NVIDIA RTX 3090 GPU, respectively. In the output decode experiment with Mamba2-2.7B, FastMamba attains 6\times higher energy efficiency than RTX 3090 GPU.