Learning Quadrupedal Locomotion for a Heavy Hydraulic Robot Using an Actuator Model

TL;DR

This paper introduces an analytical hydraulic actuator model enabling rapid simulation for RL training, leading to successful transfer of learned quadrupedal locomotion to a heavy hydraulic robot in real-world conditions.

Contribution

The paper presents a novel analytical actuator model that accurately predicts joint torques and facilitates effective sim-to-real transfer for large hydraulic quadruped robots.

Findings

Model predicts joint torques in under 1 microsecond

RL-trained policy successfully deployed on a 300+ kg hydraulic robot

First demonstration of stable RL-based locomotion transfer on a heavy hydraulic quadruped

Abstract

The simulation-to-reality (sim-to-real) transfer of large-scale hydraulic robots presents a significant challenge in robotics because of the inherent slow control response and complex fluid dynamics. The complex dynamics result from the multiple interconnected cylinder structure and the difference in fluid rates of the cylinders. These characteristics complicate detailed simulation for all joints, making it unsuitable for reinforcement learning (RL) applications. In this work, we propose an analytical actuator model driven by hydraulic dynamics to represent the complicated actuators. The model predicts joint torques for all 12 actuators in under 1 microsecond, allowing rapid processing in RL environments. We compare our model with neural network-based actuator models and demonstrate the advantages of our model in data-limited scenarios. The locomotion policy trained in RL with our model is deployed on a hydraulic quadruped robot, which is over 300 kg. This work is the first demonstration of a successful transfer of stable and robust command-tracking locomotion with RL on a heavy hydraulic quadruped robot, demonstrating advanced sim-to-real transferability.