Average-Reward Maximum Entropy Reinforcement Learning for Underactuated Double Pendulum Tasks

TL;DR

This paper introduces AR-EAPO, a novel model-free reinforcement learning algorithm combining average-reward and maximum entropy principles, to improve swing-up and stabilization tasks in underactuated double pendulum systems within simulation environments.

Contribution

It presents the AR-EAPO algorithm, a new approach that enhances performance and robustness in underactuated double pendulum tasks without complex reward engineering or system modeling.

Findings

Achieved improved performance over baseline methods.

Demonstrated robustness in simulation tasks.

Effective without heavy reward engineering.

Abstract

This report presents a solution for the swing-up and stabilisation tasks of the acrobot and the pendubot, developed for the AI Olympics competition at IROS 2024. Our approach employs the Average-Reward Entropy Advantage Policy Optimization (AR-EAPO), a model-free reinforcement learning (RL) algorithm that combines average-reward RL and maximum entropy RL. Results demonstrate that our controller achieves improved performance and robustness scores compared to established baseline methods in both the acrobot and pendubot scenarios, without the need for a heavily engineered reward function or system model. The current results are applicable exclusively to the simulation stage setup.