GlobalMamba: Global Image Serialization for Vision Mamba

TL;DR

GlobalMamba introduces a novel global image serialization method using DCT to transform images into sequences that preserve 2D structural information, enhancing vision mamba models' ability to capture global context.

Contribution

The paper proposes a new global image serialization technique using DCT, enabling vision mamba models to better exploit global information and causal relations in images.

Findings

Improved image classification accuracy on ImageNet-1K

Enhanced object detection performance on COCO

Better semantic segmentation results on ADE20K

Abstract

Vision mambas have demonstrated strong performance with linear complexity to the number of vision tokens. Their efficiency results from processing image tokens sequentially. However, most existing methods employ patch-based image tokenization and then flatten them into 1D sequences for causal processing, which ignore the intrinsic 2D structural correlations of images. It is also difficult to extract global information by sequential processing of local patches. In this paper, we propose a global image serialization method to transform the image into a sequence of causal tokens, which contain global information of the 2D image. We first convert the image from the spatial domain to the frequency domain using Discrete Cosine Transform (DCT) and then arrange the pixels with corresponding frequency ranges. We further transform each set within the same frequency band back to the spatial domain to obtain a series of images before tokenization. We construct a vision mamba model, GlobalMamba, with a causal input format based on the proposed global image serialization, which can better exploit the causal relations among image sequences. Extensive experiments demonstrate the effectiveness of our GlobalMamba, including image classification on ImageNet-1K, object detection on COCO, and semantic segmentation on ADE20K.