Revisiting Token Compression for Accelerating ViT-based Sparse Multi-View 3D Object Detectors

TL;DR

SEPatch3D introduces a dynamic patch size adjustment framework for ViT-based 3D object detection, improving inference speed and efficiency while maintaining detection accuracy.

Contribution

It proposes a novel method combining adaptive patch sizing, informative patch selection, and feature enhancement to address limitations of existing token compression strategies.

Findings

Achieves up to 57% faster inference than StreamPETR.

Provides 20% higher efficiency compared to ToC3D-faster.

Maintains comparable detection accuracy on nuScenes and Argoverse 2.

Abstract

Vision Transformer (ViT)-based sparse multi-view 3D object detectors have achieved remarkable accuracy but still suffer from high inference latency due to heavy token processing. To accelerate these models, token compression has been widely explored. However, our revisit of existing strategies, such as token pruning, merging, and patch size enlargement, reveals that they often discard informative background cues, disrupt contextual consistency, and lose fine-grained semantics, negatively affecting 3D detection. To overcome these limitations, we propose SEPatch3D, a novel framework that dynamically adjusts patch sizes while preserving critical semantic information within coarse patches. Specifically, we design Spatiotemporal-aware Patch Size Selection (SPSS) that assigns small patches to scenes containing nearby objects to preserve fine details and large patches to background-dominated scenes to reduce computation cost. To further mitigate potential detail loss, Informative Patch Selection (IPS) selects the informative patches for feature refinement, and Cross-Granularity Feature Enhancement (CGFE) injects fine-grained details into selected coarse patches, enriching semantic features. Experiments on the nuScenes and Argoverse 2 validation sets show that SEPatch3D achieves up to \textbf{57\%} faster inference than the StreamPETR baseline and \textbf{20\%} higher efficiency than the state-of-the-art ToC3D-faster, while preserving comparable detection accuracy. Code is available at https://github.com/Mingqj/SEPatch3D.