Learning to See Physical Properties with Active Sensing Motor Policies

TL;DR

This paper introduces a self-supervised approach for robots to learn physical terrain properties from images, using active sensing policies to improve estimation accuracy during real-world traversal.

Contribution

It presents Active Sensing Motor Policies (ASMP) that guide exploration to enhance physical property estimation without requiring labeled real-world data.

Findings

Robots can accurately predict terrain physical properties from visual data.

Active exploration improves the accuracy of physical property estimation.

The system generalizes from ground-level to overhead imagery.

Abstract

Knowledge of terrain's physical properties inferred from color images can aid in making efficient robotic locomotion plans. However, unlike image classification, it is unintuitive for humans to label image patches with physical properties. Without labeled data, building a vision system that takes as input the observed terrain and predicts physical properties remains challenging. We present a method that overcomes this challenge by self-supervised labeling of images captured by robots during real-world traversal with physical property estimators trained in simulation. To ensure accurate labeling, we introduce Active Sensing Motor Policies (ASMP), which are trained to explore locomotion behaviors that increase the accuracy of estimating physical parameters. For instance, the quadruped robot learns to swipe its foot against the ground to estimate the friction coefficient accurately. We show that the visual system trained with a small amount of real-world traversal data accurately predicts physical parameters. The trained system is robust and works even with overhead images captured by a drone despite being trained on data collected by cameras attached to a quadruped robot walking on the ground.