Multi-critic Learning for Whole-body End-effector Twist Tracking

TL;DR

This paper introduces a multi-critic reinforcement learning framework for simultaneous whole-body locomotion and end-effector control, effectively managing conflicting task goals and enabling smooth trajectory tracking in quadruped robots with arms.

Contribution

It proposes a novel multi-critic actor architecture and twist-based task formulation for velocity-aware, conflict-resolving whole-body control in robotics.

Findings

Effective simultaneous locomotion and manipulation demonstrated in simulation.

Emergent whole-body behaviors, including base assistance for arm extension.

Successful hardware validation on a quadruped robot.

Abstract

Learning whole-body control for locomotion and arm motions in a single policy has challenges, as the two tasks have conflicting goals. For instance, efficient locomotion typically favors a horizontal base orientation, while end-effector tracking may benefit from base tilting to extend reachability. Additionally, current Reinforcement Learning (RL) approaches using a pose-based task specification lack the ability to directly control the end-effector velocity, making smoothly executing trajectories very challenging. To address these limitations, we propose an RL-based framework that allows for dynamic, velocity-aware whole-body end-effector control. Our method introduces a multi-critic actor architecture that decouples the reward signals for locomotion and manipulation, simplifying reward tuning and allowing the policy to resolve task conflicts more effectively. Furthermore, we design a twist-based end-effector task formulation that can track both discrete poses and motion trajectories. We validate our approach through a set of simulation and hardware experiments using a quadruped robot equipped with a robotic arm. The resulting controller can simultaneously walk and move its end-effector and shows emergent whole-body behaviors, where the base assists the arm in extending the workspace, despite a lack of explicit formulations. Videos and supplementary material can be found at multi-critic-locomanipulation.github.io.