Multi-HMR: Multi-Person Whole-Body Human Mesh Recovery in a Single Shot

TL;DR

Multi-HMR is a novel single-shot model capable of recovering full-body 3D human meshes, including hands and facial expressions, from a single RGB image, using a transformer-based architecture and a new dataset for training.

Contribution

It introduces Multi-HMR, a transformer-based approach for multi-person whole-body mesh recovery, and the CUFFS dataset for improved hand pose estimation.

Findings

Achieves state-of-the-art results on whole-body benchmarks.

Incorporating CUFFS dataset improves hand pose predictions.

Fast and competitive performance with ViT-S backbone at 448x448 resolution.

Abstract

We present Multi-HMR, a strong sigle-shot model for multi-person 3D human mesh recovery from a single RGB image. Predictions encompass the whole body, i.e., including hands and facial expressions, using the SMPL-X parametric model and 3D location in the camera coordinate system. Our model detects people by predicting coarse 2D heatmaps of person locations, using features produced by a standard Vision Transformer (ViT) backbone. It then predicts their whole-body pose, shape and 3D location using a new cross-attention module called the Human Prediction Head (HPH), with one query attending to the entire set of features for each detected person. As direct prediction of fine-grained hands and facial poses in a single shot, i.e., without relying on explicit crops around body parts, is hard to learn from existing data, we introduce CUFFS, the Close-Up Frames of Full-Body Subjects dataset, containing humans close to the camera with diverse hand poses. We show that incorporating it into the training data further enhances predictions, particularly for hands. Multi-HMR also optionally accounts for camera intrinsics, if available, by encoding camera ray directions for each image token. This simple design achieves strong performance on whole-body and body-only benchmarks simultaneously: a ViT-S backbone on $448{\times}448$ images already yields a fast and competitive model, while larger models and higher resolutions obtain state-of-the-art results.