Fourier Compressor: Frequency-Domain Visual Token Compression for Vision-Language Models

TL;DR

Fourier Compressor is a novel, parameter-free frequency-domain module that significantly reduces computational costs in vision-language models while maintaining high accuracy, applicable to images and videos.

Contribution

It introduces a frequency-domain visual token compression method that outperforms existing parameter-free approaches and generalizes across multiple architectures and tasks.

Findings

Retains over 96% of original accuracy with up to 83.8% FLOPs reduction.

Boosts generation speed by 31.2%.

Outperforms existing parameter-free methods and surpasses some parameterized approaches.

Abstract

Vision-Language Models (VLMs) incur substantial computational overhead and inference latency due to the large number of vision tokens introduced by high-resolution image and video inputs. Existing parameter-free token compression methods typically rely on token selection or merging, yet they risk discarding substantial visual information or distorting the original representation distribution, resulting in pronounced performance degradation at high compression ratios. In response, we aim to explore a more effective and efficient visual token compression strategy, with a promising direction in the frequency domain. Motivated by the success of frequency-domain transforms in image compression (e.g., JPEG), we systematically analyze the frequency redundancy in visual representations and uncover a non-uniform distribution of semantic information across frequency bands. Building upon this, we introduce Fourier Compressor, an effective, parameter-free, and highly generalizable module that removes redundancy from visual representations within the frequency domain. Implemented via FFT with $\mathcal{O}(n^2 \log n)$ complexity and no additional parameters, Fourier Compressor introduces negligible computational overhead while preserving semantic fidelity. Extensive experiments on image-based benchmarks demonstrate that our method achieves a favorable performance-efficiency trade-off, retaining over 96% of the original accuracy while reducing inference FLOPs by up to 83.8% and boosting generation speed by 31.2%. It consistently outperforms existing parameter-free methods and even surpasses some parameterized approaches. Importantly, Fourier Compressor generalizes consistently across both LLaVA and Qwen-VL architectures, and further extends to video understanding tasks, highlighting its practical applicability for efficient VLMs.