Understanding Particles From Video: Property Estimation of Granular Materials via Visuo-Haptic Learning

TL;DR

This paper presents a visuo-haptic learning method to estimate particle size and density of granular materials from video, reducing the need for dedicated equipment and human effort, with validated experiments and real-world application.

Contribution

A novel visuo-haptic learning framework that estimates granular material properties solely from visual data, leveraging a contact model-inspired neural network.

Findings

Accurately estimates particle size and density from video data.

Demonstrates strong correlation between visual-haptic data and particle properties.

Validates generalization and real-world applicability through extensive experiments.

Abstract

Granular materials (GMs) are ubiquitous in daily life. Understanding their properties is also important, especially in agriculture and industry. However, existing works require dedicated measurement equipment and also need large human efforts to handle a large number of particles. In this paper, we introduce a method for estimating the relative values of particle size and density from the video of the interaction with GMs. It is trained on a visuo-haptic learning framework inspired by a contact model, which reveals the strong correlation between GM properties and the visual-haptic data during the probe-dragging in the GMs. After training, the network can map the visual modality well to the haptic signal and implicitly characterize the relative distribution of particle properties in its latent embeddings, as interpreted in that contact model. Therefore, we can analyze GM properties using the trained encoder, and only visual information is needed without extra sensory modalities and human efforts for labeling. The presented GM property estimator has been extensively validated via comparison and ablation experiments. The generalization capability has also been evaluated and a real-world application on the beach is also demonstrated. Experiment videos are available at \url{https://sites.google.com/view/gmwork/vhlearning} .