Physically Guided Visual Mass Estimation from a Single RGB Image

TL;DR

This paper introduces a physically structured framework for estimating object mass from a single RGB image by combining geometry, semantics, and appearance cues, outperforming existing methods.

Contribution

It presents a novel approach that aligns visual cues with physical factors like volume and density, using monocular depth and vision-language models for improved mass estimation.

Findings

Outperforms state-of-the-art methods on image2mass and ABO-500 datasets.

Effectively combines geometry, semantics, and appearance for mass prediction.

Uses physically guided latent factors with mass-only supervision.

Abstract

Estimating object mass from visual input is challenging because mass depends jointly on geometric volume and material-dependent density, neither of which is directly observable from RGB appearance. Consequently, mass prediction from pixels is ill-posed and therefore benefits from physically meaningful representations to constrain the space of plausible solutions. We propose a physically structured framework for single-image mass estimation that addresses this ambiguity by aligning visual cues with the physical factors governing mass. From a single RGB image, we recover object-centric three-dimensional geometry via monocular depth estimation to inform volume and extract coarse material semantics using a vision-language model to guide density-related reasoning. These geometry, semantic, and appearance representations are fused through an instance-adaptive gating mechanism, and two physically guided latent factors (volume- and density-related) are predicted through separate regression heads under mass-only supervision. Experiments on image2mass and ABO-500 show that the proposed method consistently outperforms state-of-the-art methods.