Estimating Object Physical Properties from RGB-D Vision and Depth Robot Sensors Using Deep Learning

TL;DR

This paper introduces a deep learning method that combines RGB and depth data from sensors to accurately estimate object mass, using synthetic datasets for training and outperforming existing benchmarks.

Contribution

It presents a novel approach integrating RGB-D data and synthetic dataset generation for improved object mass estimation in robotics.

Findings

Significantly outperforms existing benchmarks in mass estimation accuracy.

Uses synthetic RGB-D data to train models effectively.

Combines point-cloud and RGB data for enhanced estimation.

Abstract

Inertial mass plays a crucial role in robotic applications such as object grasping, manipulation, and simulation, providing a strong prior for planning and control. Accurately estimating an object's mass before interaction can significantly enhance the performance of various robotic tasks. However, mass estimation using only vision sensors is a relatively underexplored area. This paper proposes a novel approach combining sparse point-cloud data from depth images with RGB images to estimate the mass of objects. We evaluate a range of point-cloud processing architectures, alongside RGB-only methods. To overcome the limited availability of training data, we create a synthetic dataset using ShapeNetSem 3D models, simulating RGBD images via a Kinect camera. This synthetic data is used to train an image generation model for estimating dense depth maps, which we then use to augment an existing dataset of images paired with mass values. Our approach significantly outperforms existing benchmarks across all evaluated metrics. The data generation (https://github.com/RavineWindteer/ShapenetSem-to-RGBD) as well as the training of the depth estimator (https://github.com/RavineWindteer/GLPDepth-Edited) and the mass estimator (https://github.com/RavineWindteer/Depth-mass-estimator) are available online.