Tracking Control for a Spherical Pendulum via Curriculum Reinforcement Learning

TL;DR

This paper introduces a curriculum reinforcement learning approach to efficiently learn a tracking controller for a spherical pendulum on a robotic arm, achieving robust real-world performance by leveraging structured learning and parallelized simulations.

Contribution

The paper presents an automatic curriculum generation method integrated with RL for non-linear tracking control, improving learning speed and robustness over standard RL methods.

Findings

Faster convergence compared to baseline RL methods.

Achieved real-system performance matching optimal control.

Demonstrated effective joint learning of state estimation and control.

Abstract

Reinforcement Learning (RL) allows learning non-trivial robot control laws purely from data. However, many successful applications of RL have relied on ad-hoc regularizations, such as hand-crafted curricula, to regularize the learning performance. In this paper, we pair a recent algorithm for automatically building curricula with RL on massively parallelized simulations to learn a tracking controller for a spherical pendulum on a robotic arm via RL. Through an improved optimization scheme that better respects the non-Euclidean task structure, we allow the method to reliably generate curricula of trajectories to be tracked, resulting in faster and more robust learning compared to an RL baseline that does not exploit this form of structured learning. The learned policy matches the performance of an optimal control baseline on the real system, demonstrating the potential of curriculum RL to jointly learn state estimation and control for non-linear tracking tasks.