Improving Reinforcement Learning Efficiency with Auxiliary Tasks in Non-Visual Environments: A Comparison

TL;DR

This paper compares auxiliary tasks for decoupled representation learning in non-visual RL environments, showing that they improve performance mainly in complex tasks and that learning environment dynamics is more effective than reward prediction.

Contribution

It provides a systematic comparison of auxiliary tasks in decoupled representation learning for low-dimensional non-visual RL environments, highlighting when they are beneficial.

Findings

Auxiliary tasks improve RL performance mainly in complex environments.

Learning environment dynamics outperforms reward prediction as an auxiliary task.

Representation learning benefits are limited in simple or low-complexity environments.

Abstract

Real-world reinforcement learning (RL) environments, whether in robotics or industrial settings, often involve non-visual observations and require not only efficient but also reliable and thus interpretable and flexible RL approaches. To improve efficiency, agents that perform state representation learning with auxiliary tasks have been widely studied in visual observation contexts. However, for real-world problems, dedicated representation learning modules that are decoupled from RL agents are more suited to meet requirements. This study compares common auxiliary tasks based on, to the best of our knowledge, the only decoupled representation learning method for low-dimensional non-visual observations. We evaluate potential improvements in sample efficiency and returns for environments ranging from a simple pendulum to a complex simulated robotics task. Our findings show that representation learning with auxiliary tasks only provides performance gains in sufficiently complex environments and that learning environment dynamics is preferable to predicting rewards. These insights can inform future development of interpretable representation learning approaches for non-visual observations and advance the use of RL solutions in real-world scenarios.