Investigating Anthropometric Fidelity in SAM 3D Body

TL;DR

This paper analyzes the limitations of SAM 3D Body in capturing detailed anthropometric deviations, attributing it to architectural biases and proposing future pathways for high-precision medical applications.

Contribution

It identifies the perception-distortion trade-off as a cause for smoothing biological details and suggests hybrid representations and Medical-in-the-Loop alignment for improvement.

Findings

SAM 3D Body struggles with detailed anthropometric deviations.

The model exhibits a 'regression to the mean' effect due to its architecture.

Proposed pathways include hybrid representations and Medical-in-the-Loop alignment.

Abstract

The release of SAM 3D Body is a recent development in human mesh recovery, demonstrating improved performance in producing clean, topologically coherent meshes from single images. By leveraging the Momentum Human Rig (MHR), it achieves robustness to occlusion and diverse poses. However, our evaluation reveals a specific and consistent limitation: the model struggles to reconstruct detailed anthropometric deviations, particularly in populations exhibiting distinctive morphological alterations such as geriatric muscle atrophy, scoliosis, or pregnancy, even when these features are prominent in the input image. In this paper, we investigate this phenomenon not as a failure of the model's capacity, but as a byproduct of the "perception-distortion trade-off". We posit that the architectural reliance on the low-dimensional parametric MHR representation, combined with semantic-invariant conditioning (DINOv3) and annotation-based alignment, creates a pervasive "regression to the mean" effect. We analyze these mechanisms to understand why individual biological details are smoothed out. Furthermore, we state our contributions by proposing specific, constructive pathways for future work, such as implicit-explicit hybrid representations and Medical-in-the-Loop alignment, to extend the baseline performance of SAM 3D Body into the high-precision medical domain.