Visual Manipulation with Legs

TL;DR

This paper presents a novel system enabling quadruped robots to manipulate objects using their legs, combining visual RL policies and loco-manipulation control for versatile non-prehensile interactions in simulation and real-world tests.

Contribution

It introduces a combined visual manipulation and loco-manipulator system for leg-based object interaction, inspired by animal limb versatility, with real-world validation.

Findings

Enhanced object manipulation capabilities with quadruped legs.

Successful real-world implementation of leg-based manipulation.

Improved versatility over previous methods.

Abstract

Animals use limbs for both locomotion and manipulation. We aim to equip quadruped robots with similar versatility. This work introduces a system that enables quadruped robots to interact with objects using their legs, inspired by non-prehensile manipulation. The system has two main components: a visual manipulation policy module and a loco-manipulator module. The visual manipulation policy, trained with reinforcement learning (RL) using point cloud observations and object-centric actions, decides how the leg should interact with the object. The loco-manipulator controller manages leg movements and body pose adjustments, based on impedance control and Model Predictive Control (MPC). Besides manipulating objects with a single leg, the system can select from the left or right leg based on critic maps and move objects to distant goals through base adjustment. Experiments evaluate the system on object pose alignment tasks in both simulation and the real world, demonstrating more versatile object manipulation skills with legs than previous work. Videos can be found at https://legged-manipulation.github.io/