DeforHMR: Vision Transformer with Deformable Cross-Attention for 3D Human Mesh Recovery

TL;DR

DeforHMR introduces a deformable cross-attention transformer framework that significantly improves 3D human mesh recovery from single images by effectively leveraging pretrained vision transformers.

Contribution

The paper proposes a novel deformable cross-attention mechanism within a transformer decoder for enhanced 3D human pose prediction, advancing the state-of-the-art in monocular HMR.

Findings

Achieves state-of-the-art results on 3DPW and RICH benchmarks.

Effectively leverages pretrained vision transformers for 3D human mesh recovery.

Introduces a flexible, query-agnostic deformable attention mechanism.

Abstract

Human Mesh Recovery (HMR) is an important yet challenging problem with applications across various domains including motion capture, augmented reality, and biomechanics. Accurately predicting human pose parameters from a single image remains a challenging 3D computer vision task. In this work, we introduce DeforHMR, a novel regression-based monocular HMR framework designed to enhance the prediction of human pose parameters using deformable attention transformers. DeforHMR leverages a novel query-agnostic deformable cross-attention mechanism within the transformer decoder to effectively regress the visual features extracted from a frozen pretrained vision transformer (ViT) encoder. The proposed deformable cross-attention mechanism allows the model to attend to relevant spatial features more flexibly and in a data-dependent manner. Equipped with a transformer decoder capable of spatially-nuanced attention, DeforHMR achieves state-of-the-art performance for single-frame regression-based methods on the widely used 3D HMR benchmarks 3DPW and RICH. By pushing the boundary on the field of 3D human mesh recovery through deformable attention, we introduce an new, effective paradigm for decoding local spatial information from large pretrained vision encoders in computer vision.