VGGT-HPE: Reframing Head Pose Estimation as Relative Pose Prediction

TL;DR

VGGT-HPE introduces a relative head pose estimation method that predicts transformations between configurations, trained solely on synthetic data, achieving state-of-the-art results without real-world training.

Contribution

The paper proposes a novel relative head pose estimation approach using synthetic data, outperforming traditional absolute regression methods and validating the advantages of relative prediction.

Findings

Achieves state-of-the-art results on BIWI benchmark.

Relative prediction outperforms absolute regression, especially on difficult poses.

Zero real-world training data suffices for high accuracy.

Abstract

Monocular head pose estimation is traditionally formulated as direct regression from a single image to an absolute pose. This paradigm forces the network to implicitly internalize a dataset-specific canonical reference frame. In this work, we argue that predicting the relative rigid transformation between two observed head configurations is a fundamentally easier and more robust formulation. We introduce VGGT-HPE, a relative head pose estimator built upon a general-purpose geometry foundation model. Finetuned exclusively on synthetic facial renderings, our method sidesteps the need for an implicit anchor by reducing the problem to estimating a geometric displacement from an explicitly provided anchor with a known pose. As a practical benefit, the relative formulation also allows the anchor to be chosen at test time - for instance, a near-neutral frame or a temporally adjacent one - so that the prediction difficulty can be controlled by the application. Despite zero real-world training data, VGGT-HPE achieves state-of-the-art results on the BIWI benchmark, outperforming established absolute regression methods trained on mixed and real datasets. Through controlled easy- and hard-pair benchmarks, we also systematically validate our core hypothesis: relative prediction is intrinsically more accurate than absolute regression, with the advantage scaling alongside the difficulty of the target pose. Project page and code: https://vasilikivas.github.io/VGGT-HPE