AVGGT: Rethinking Global Attention for Accelerating VGGT

TL;DR

This paper analyzes the role of global attention in VGGT and $ ext{pi}^3$ models, revealing their layered functions, and proposes a training-free acceleration scheme that significantly speeds up inference while maintaining accuracy.

Contribution

The paper provides a systematic analysis of global attention modules in multi-view 3D models and introduces a novel, training-free acceleration method based on insights from this analysis.

Findings

Achieves 2x to 10x inference speedup across different frame lengths.

Maintains or slightly improves model accuracy compared to original models.

Robustly performs in dense multi-view settings where sparse attention baselines fail.

Abstract

Models such as VGGT and $\pi^3$ have shown strong multi-view 3D performance, but their heavy reliance on global self-attention results in high computational cost. Existing sparse-attention variants offer partial speedups, yet lack a systematic analysis of how global attention contributes to multi-view reasoning. In this paper, we first conduct an in-depth investigation of the global attention modules in VGGT and $\pi^3$ to better understand their roles. Our analysis reveals a clear division of roles in the alternating global-frame architecture: early global layers do not form meaningful correspondences, middle layers perform cross-view alignment, and last layers provide only minor refinements. Guided by these findings, we propose a training-free two-step acceleration scheme: (1) converting early global layers into frame attention, and (2) subsampling global attention by subsampling K/V over patch tokens with diagonal preservation and a mean-fill component. We instantiate this strategy on VGGT and $\pi^3$ and evaluate across standard pose and point-map benchmarks. Our method achieves substantial inference acceleration across different context lengths, yielding about $2\times$ speedup at 100 frames, $4$--$5\times$ at 300 frames, and $8$--$10\times$ at 800 frames, while matching or slightly improving the accuracy of the original models and remaining robust in extremely dense multi-view settings where prior sparse-attention baselines fail.