HybridTM: Combining Transformer and Mamba for 3D Semantic Segmentation

TL;DR

HybridTM innovatively combines Transformer and Mamba architectures with an inner layer hybrid strategy to enhance 3D semantic segmentation, achieving state-of-the-art results on multiple benchmarks.

Contribution

First hybrid architecture integrating Transformer and Mamba for 3D segmentation with a novel inner layer hybrid strategy.

Findings

Achieves state-of-the-art performance on ScanNet, ScanNet200, and nuScenes.

Demonstrates superior generalization across diverse datasets.

Effectively captures both long-range dependencies and local features.

Abstract

Transformer-based methods have demonstrated remarkable capabilities in 3D semantic segmentation through their powerful attention mechanisms, but the quadratic complexity limits their modeling of long-range dependencies in large-scale point clouds. While recent Mamba-based approaches offer efficient processing with linear complexity, they struggle with feature representation when extracting 3D features. However, effectively combining these complementary strengths remains an open challenge in this field. In this paper, we propose HybridTM, the first hybrid architecture that integrates Transformer and Mamba for 3D semantic segmentation. In addition, we propose the Inner Layer Hybrid Strategy, which combines attention and Mamba at a finer granularity, enabling simultaneous capture of long-range dependencies and fine-grained local features. Extensive experiments demonstrate the effectiveness and generalization of our HybridTM on diverse indoor and outdoor datasets. Furthermore, our HybridTM achieves state-of-the-art performance on ScanNet, ScanNet200, and nuScenes benchmarks. The code will be made available at https://github.com/deepinact/HybridTM.