# Graph-Based and Multi-Stage Constraints for Hand–Object Reconstruction

## TL;DR

This paper introduces a new framework for reconstructing hand-object interactions using a multi-stage approach that improves physical accuracy and handles occlusions better than existing methods.

## Contribution

The novel framework uses progressive physical consistency and geometric accuracy through a multi-stage optimization pipeline with mutual attention and SDF-based contact modeling.

## Key findings

- The proposed method outperforms state-of-the-art techniques on the ObMan and DexYCB benchmarks.
- On ObMan, the method achieves hand and object metrics of 0.077 cm² and 0.483 cm², respectively.
- On DexYCB, it achieves hand and object metrics of 0.251 cm² and 1.127 cm², respectively.

## Abstract

Reconstructing hand and object shapes from a single view during interaction remains challenging due to severe mutual occlusion and the need for high physical plausibility. To address this, we propose a novel framework for hand–object interaction reconstruction based on holistic, multi-stage collaborative optimization. Unlike methods that process hands and objects independently or apply constraints as late-stage post-processing, our model progressively enforces physical consistency and geometric accuracy throughout the entire reconstruction pipeline. Our network takes an RGB-D image as input. An adaptive feature fusion module first combines color and depth information to improve robustness against sensing uncertainties. We then introduce structural priors for 2D pose estimation and leverage texture cues to refine depth-based 3D pose initialization. Central to our approach is the iterative application of a dense mutual attention mechanism during sparse-to-dense mesh recovery, which dynamically captures interaction dependencies while refining geometry. Finally, we use a Signed Distance Function (SDF) representation explicitly designed for contact surfaces to prevent interpenetration and ensure physically plausible results. Through comprehensive experiments, our method demonstrates significant improvements on the challenging ObMan and DexYCB benchmarks, outperforming state-of-the-art techniques. Specifically, on the ObMan dataset, our approach achieves hand CDh and object CDo metrics of 0.077 cm2 and 0.483 cm2, respectively. Similarly, on the DexYCB dataset, it attains hand CDh and object CDo values of 0.251 cm2 and 1.127 cm2, respectively.