Finetuning Deep Reinforcement Learning Policies with Evolutionary Strategies for Control of Underactuated Robots

TL;DR

This paper presents a method combining Deep Reinforcement Learning with Evolutionary Strategies to fine-tune control policies for underactuated robots, resulting in improved performance and robustness in complex tasks.

Contribution

It introduces a novel approach that refines Deep RL policies using Evolutionary Strategies, specifically targeting underactuated robotic control for enhanced effectiveness.

Findings

Significant performance improvements over baseline policies.

Enhanced robustness of control strategies.

Competitive scores achieved in AI Olympics tasks.

Abstract

Deep Reinforcement Learning (RL) has emerged as a powerful method for addressing complex control problems, particularly those involving underactuated robotic systems. However, in some cases, policies may require refinement to achieve optimal performance and robustness aligned with specific task objectives. In this paper, we propose an approach for fine-tuning Deep RL policies using Evolutionary Strategies (ES) to enhance control performance for underactuated robots. Our method involves initially training an RL agent with Soft-Actor Critic (SAC) using a surrogate reward function designed to approximate complex specific scoring metrics. We subsequently refine this learned policy through a zero-order optimization step employing the Separable Natural Evolution Strategy (SNES), directly targeting the original score. Experimental evaluations conducted in the context of the 2nd AI Olympics with RealAIGym at IROS 2024 demonstrate that our evolutionary fine-tuning significantly improves agent performance while maintaining high robustness. The resulting controllers outperform established baselines, achieving competitive scores for the competition tasks.