Analyzing the Mechanism of Attention Collapse in VGGT from a Dynamics Perspective

TL;DR

This paper provides a mathematical analysis of attention collapse in VGGT, revealing how token features converge to a degenerate state and proposing insights for improving scalable 3D-vision transformers.

Contribution

It introduces a mean-field PDE model that predicts attention collapse dynamics and explains the effectiveness of token-merging remedies in VGGT.

Findings

Attention matrices become near rank-one with many frames.

Token features converge to a Dirac-type measure at a rate of O(1/L).

Token-merging delays collapse by reducing the diffusion coefficient.

Abstract

Visual Geometry Grounded Transformer (VGGT) delivers state-of-the-art feed-forward 3D reconstruction, yet its global self-attention layer suffers from a drastic collapse phenomenon when the input sequence exceeds a few hundred frames: attention matrices rapidly become near rank-one, token geometry degenerates to an almost one-dimensional subspace, and reconstruction error accumulates super-linearly.In this report,we establish a rigorous mathematical explanation of the collapse by viewing the global-attention iteration as a degenerate diffusion process.We prove that,in VGGT, the token-feature flow converges toward a Dirac-type measure at a $O(1/L)$ rate, where $L$ is the layer index, yielding a closed-form mean-field partial differential equation that precisely predicts the empirically observed rank profile.The theory quantitatively matches the attention-heat-map evolution and a series of experiments outcomes reported in relevant works and explains why its token-merging remedy -- which periodically removes redundant tokens -- slows the effective diffusion coefficient and thereby delays collapse without additional training.We believe the analysis provides a principled lens for interpreting future scalable 3D-vision transformers,and we highlight its potential for multi-modal generalization.