Visual-Tactile Sensing for Real-time Liquid Volume Estimation in Grasping

TL;DR

This paper introduces a deep visuo-tactile model for real-time liquid volume estimation inside deformable containers, enabling precise robotic manipulation without extra sensors.

Contribution

It develops an end-to-end multi-modal convolutional network for high-precision volume estimation and integrates it into a robotic system for adaptive grasping and control.

Findings

Achieved around 2 ml error in volume estimation.

Validated the model's effectiveness on a robotic platform.

Demonstrated real-time adaptive grasping based on predictions.

Abstract

We propose a deep visuo-tactile model for realtime estimation of the liquid inside a deformable container in a proprioceptive way.We fuse two sensory modalities, i.e., the raw visual inputs from the RGB camera and the tactile cues from our specific tactile sensor without any extra sensor calibrations.The robotic system is well controlled and adjusted based on the estimation model in real time. The main contributions and novelties of our work are listed as follows: 1) Explore a proprioceptive way for liquid volume estimation by developing an end-to-end predictive model with multi-modal convolutional networks, which achieve a high precision with an error of around 2 ml in the experimental validation. 2) Propose a multi-task learning architecture which comprehensively considers the losses from both classification and regression tasks, and comparatively evaluate the performance of each variant on the collected data and actual robotic platform. 3) Utilize the proprioceptive robotic system to accurately serve and control the requested volume of liquid, which is continuously flowing into a deformable container in real time. 4) Adaptively adjust the grasping plan to achieve more stable grasping and manipulation according to the real-time liquid volume prediction.