HeSS: Head Sensitivity Score for Sparsity Redistribution in VGGT

TL;DR

This paper introduces HeSS, a novel metric for measuring head-wise sensitivity in VGGT models, enabling targeted sparsification that maintains accuracy while reducing computational costs.

Contribution

We propose a two-stage sparsification method using HeSS to allocate attention sparsity based on head sensitivity, improving efficiency without significant accuracy loss.

Findings

HeSS accurately captures head-wise sparsification sensitivity.

Attention heads exhibit heterogeneous sensitivity characteristics.

Our method reduces performance degradation at high sparsity levels.

Abstract

Visual Geometry Grounded Transformer (VGGT) has advanced 3D vision, yet its global attention layers suffer from quadratic computational costs that hinder scalability. Several sparsification-based acceleration techniques have been proposed to alleviate this issue, but they often suffer from substantial accuracy degradation. We hypothesize that the accuracy degradation stems from the heterogeneity in head-wise sparsification sensitivity, as the existing methods apply a uniform sparsity pattern across all heads. Motivated by this hypothesis, we present a two-stage sparsification pipeline that effectively quantifies and exploits headwise sparsification sensitivity. In the first stage, we measure head-wise sparsification sensitivity using a novel metric, the Head Sensitivity Score (HeSS), which approximates the Hessian with respect to two distinct error terms on a small calibration set. In the inference stage, we perform HeSS-Guided Sparsification, leveraging the pre-computed HeSS to reallocate the total attention budget-assigning denser attention to sensitive heads and sparser attention to more robust ones. We demonstrate that HeSS effectively captures head-wise sparsification sensitivity and empirically confirm that attention heads in the global attention layers exhibit heterogeneous sensitivity characteristics. Extensive experiments further show that our method effectively mitigates performance degradation under high sparsity, demonstrating strong robustness across varying sparsification levels. Code is available at https://github.com/libary753/HeSS.