Spectral Graphormer: Spectral Graph-based Transformer for Egocentric Two-Hand Reconstruction using Multi-View Color Images

TL;DR

This paper introduces a spectral graph-based transformer framework for high-fidelity two-hand reconstruction from multi-view egocentric RGB images, addressing challenges in dataset scarcity and physical plausibility.

Contribution

It presents a novel spectral graph convolution decoder and a multi-view feature fusion strategy, enabling realistic and real-time two-hand pose estimation from egocentric images.

Findings

Effective reconstruction of two hands with high fidelity.

Good generalization from synthetic to real data.

Real-time performance suitable for AR/VR applications.

Abstract

We propose a novel transformer-based framework that reconstructs two high fidelity hands from multi-view RGB images. Unlike existing hand pose estimation methods, where one typically trains a deep network to regress hand model parameters from single RGB image, we consider a more challenging problem setting where we directly regress the absolute root poses of two-hands with extended forearm at high resolution from egocentric view. As existing datasets are either infeasible for egocentric viewpoints or lack background variations, we create a large-scale synthetic dataset with diverse scenarios and collect a real dataset from multi-calibrated camera setup to verify our proposed multi-view image feature fusion strategy. To make the reconstruction physically plausible, we propose two strategies: (i) a coarse-to-fine spectral graph convolution decoder to smoothen the meshes during upsampling and (ii) an optimisation-based refinement stage at inference to prevent self-penetrations. Through extensive quantitative and qualitative evaluations, we show that our framework is able to produce realistic two-hand reconstructions and demonstrate the generalisation of synthetic-trained models to real data, as well as real-time AR/VR applications.