Multi-action Tangled Program Graphs for Multi-task Reinforcement Learning with Continuous Control

TL;DR

This paper introduces Multi-Action TPG, a genetic programming approach for multi-task continuous reinforcement learning, validated on a new MuJoCo Half Cheetah benchmark with multiple obstacles, demonstrating superior performance and interpretability.

Contribution

The paper proposes Multi-Action TPG, extending TPG for continuous multi-task RL, and introduces a new benchmark to evaluate multi-task learning in continuous control environments.

Findings

MATPG achieves superior results with lexicase selection.

The new benchmark effectively tests multi-task RL capabilities.

The evolved graph's decision flow is fully interpretable.

Abstract

Over the past few decades, machine learning has been widely used to learn complex tasks. Reinforcement Learning (RL), inspired by human behavior, is a great example, as it involves developing specific behaviours for specific tasks. To further challenge algorithms, Multi-Task RL (MTRL) environments have been introduced, requiring a single model to learn multiple behaviors. The Tangled Program Graph (TPG) algorithm is a Genetic Programming (GP) algorithm designed for discrete MTRL environments. Recently, the MAPLE algorithm has been proposed, as another GP algorithm that achieves high results in single task continuous RL environments. A variation of the TPG is proposed alongside MAPLE, named Multi-Action TPG (MATPG) that aggregates MAPLE agents, and creates a control flow to activate them. Initially tested on single task RL environments only, MATPG achieved similar results to MAPLE. In this work, we present a new benchmark based on the MuJoCo Half Cheetah from Gymnasium. This benchmark features five distinct obstacles that are randomly positioned in front of the agent, each of which demands a unique behavior. This benchmark serves as a use case for MATPG, to prove its ability as a GP solution for continuous MTRL environments. Our experiments demonstrate its superiority in this multi-task use case when combined with lexicase selection. Furthermore, we examine the interpretability of the evolved graph, revealing that the decision flow of the model is fully interpretable.