Not All Tasks Quantize Equally: Fisher-Guided Quantization for Visual Geometry Transformer

TL;DR

This paper introduces Fisher-Guided Quantization (FGQ), a novel method that uses Fisher information to adaptively calibrate 3D vision transformer models, significantly reducing quantization errors across multiple tasks.

Contribution

FGQ leverages Fisher information to quantify task-specific sensitivities, enabling more effective calibration of 3D models for lower-bit quantization.

Findings

FGQ outperforms state-of-the-art baselines on VGGT models.

Achieves up to 39% relative improvement under 4-bit quantization.

Effectively preserves accuracy across multiple geometric tasks.

Abstract

Feed-forward 3D reconstruction models, represented by Visual Geometry Grounded Transformer (VGGT), jointly predict multiple visual geometry tasks such as depth estimation, camera pose prediction, and point cloud reconstruction in a single forward pass. They have been widely adopted in 3D vision applications, but their billion-scale parameters bring substantial memory and computation overhead, posing challenges for on-device deployment. Post-Training Quantization (PTQ) is an effective technique to reduce this overhead. Existing PTQ methods for feed-forward 3D models mainly focus on handling heavy-tailed activation distributions and constructing diverse calibration datasets. However, we observe that feed-forward 3D models predict multiple geometric attributes through a shared backbone, where different transformer blocks and hidden channels contribute distinctly to each task, resulting in substantially different sensitivities to quantization errors across tasks, blocks, and channels. Consequently, treating all tasks equally over-emphasizes insensitive tasks and causes significant accuracy loss on the sensitive ones. To address this issue, we propose Fisher-Guided Quantization (FGQ) for feed-forward 3D reconstruction models. Specifically, FGQ uses the diagonal Fisher information matrix to quantify the different sensitivities across tasks, blocks, and channels, and incorporates these sensitivities into the Learnable Affine Transformation during calibration to better preserve the channels and blocks most critical to each task. Extensive experiments across camera pose estimation, point map reconstruction, and depth estimation show that FGQ consistently outperforms state-of-the-art quantization baselines on VGGT, achieving up to 39% relative improvement under the 4-bit quantization.