PaceVGGT: Pre-Alternating-Attention Token Pruning for Visual Geometry Transformers

TL;DR

PaceVGGT introduces a pre-attention token pruning method for VGGT models, significantly reducing inference latency while maintaining reconstruction quality in 3D tasks.

Contribution

It proposes a novel pre-attention token pruning framework with a lightweight scorer and feature-guided restoration, enabling faster VGGT inference without quality loss.

Findings

Reduces ScanNet-50 latency by 5.1x at N=300 tokens.

Achieves 1.47x latency reduction over LiteVGGT at N=1000.

Maintains reconstruction quality while accelerating inference.

Abstract

Visual Geometry Transformer (VGGT) is a strong feed-forward model for multiple 3D tasks, but its Alternating-Attention (AA) stack scales quadratically in the total token count, making long clips expensive. Existing token-reduction accelerators operate inside AA, leaving the patch grid that enters AA uncompressed. We introduce PaceVGGT, a pre-AA token pruning framework that prunes DINO patch tokens before the first AA block of a frozen VGGT. PaceVGGT trains a lightweight Token Scorer that estimates per-token importance from DINO features. The scorer is first distilled against an AA-internal attention target from the unpruned backbone, then refined under downstream camera, depth, and point-map losses. A per-frame keep budget fixes the backbone-visible sequence length, while an importance-adaptive merge/prune assignment preserves residual content from high-saliency frames under a fixed total merge budget. A Feature-guided Restoration module reconstructs the dense spatial grid required by the prediction heads. On ScanNet-50 and 7-Scenes, PaceVGGT remains on the reconstruction quality--latency frontier while reducing inference latency. On ScanNet-50, it reduces latency by \(5.1\times\) over unmodified VGGT at \(N=300\) and \(1.47\times\) over LiteVGGT at \(N=1000\). These results identify pre-AA pruning as a viable acceleration route for frozen VGGT-style geometry transformers.