SAT-HMR: Real-Time Multi-Person 3D Mesh Estimation via Scale-Adaptive Tokens

TL;DR

SAT-HMR introduces scale-adaptive tokens within a one-stage DETR-based framework to efficiently estimate multiple 3D human meshes from a single RGB image in real-time, balancing accuracy and computational cost.

Contribution

The paper presents a novel scale-adaptive token mechanism that dynamically adjusts processing resolution based on individual scale, improving efficiency in multi-person 3D mesh estimation.

Findings

Achieves real-time inference with competitive accuracy.

Reduces computational overhead compared to high-resolution methods.

Effectively handles varying scales of individuals in images.

Abstract

We propose a one-stage framework for real-time multi-person 3D human mesh estimation from a single RGB image. While current one-stage methods, which follow a DETR-style pipeline, achieve state-of-the-art (SOTA) performance with high-resolution inputs, we observe that this particularly benefits the estimation of individuals in smaller scales of the image (e.g., those far from the camera), but at the cost of significantly increased computation overhead. To address this, we introduce scale-adaptive tokens that are dynamically adjusted based on the relative scale of each individual in the image within the DETR framework. Specifically, individuals in smaller scales are processed at higher resolutions, larger ones at lower resolutions, and background regions are further distilled. These scale-adaptive tokens more efficiently encode the image features, facilitating subsequent decoding to regress the human mesh, while allowing the model to allocate computational resources more effectively and focus on more challenging cases. Experiments show that our method preserves the accuracy benefits of high-resolution processing while substantially reducing computational cost, achieving real-time inference with performance comparable to SOTA methods.