SP$^2$T: Sparse Proxy Attention for Dual-stream Point Transformer

TL;DR

SP$^2$T introduces a novel dual-stream point transformer with sparse proxy attention, improving 3D point cloud understanding by balancing local-global features efficiently and achieving state-of-the-art results.

Contribution

The paper proposes Sparse Proxy Point Transformer (SP$^2$T), featuring spatial proxy sampling, sparse proxy attention, and a dual-stream architecture for enhanced 3D understanding.

Findings

Achieves state-of-the-art results on indoor and outdoor benchmarks.

Improves mIoU by +3.8% on S3DIS and +22.9% on Sem.KITTI.

Demonstrates efficient proxy interaction with table-based relative bias.

Abstract

Point transformers have demonstrated remarkable progress in 3D understanding through expanded receptive fields (RF), but further expanding the RF leads to dilution in group attention and decreases detailed feature extraction capability. Proxy, which serves as abstract representations for simplifying feature maps, enables global RF. However, existing proxy-based approaches face critical limitations: Global proxies incur quadratic complexity for large-scale point clouds and suffer positional ambiguity, while local proxy alternatives struggle with 1) Unreliable sampling from the geometrically diverse point cloud, 2) Inefficient proxy interaction computation, and 3) Imbalanced local-global information fusion; To address these challenges, we propose Sparse Proxy Point Transformer (SP$^{2}$T) -- a local proxy-based dual-stream point transformer with three key innovations: First, for reliable sampling, spatial-wise proxy sampling with vertex-based associations enables robust sampling on geometrically diverse point clouds. Second, for efficient proxy interaction, sparse proxy attention with a table-based relative bias effectively achieves the interaction with efficient map-reduce computation. Third, for local-global information fusion, our dual-stream architecture maintains local-global balance through parallel branches. Comprehensive experiments reveal that SP$^{2}$T sets state-of-the-art results with acceptable latency on indoor and outdoor 3D comprehension benchmarks, demonstrating marked improvement (+3.8% mIoU vs. SPoTr@S3DIS, +22.9% mIoU vs. PointASNL@Sem.KITTI) compared to other proxy-based point cloud methods.