Investigating the Impact of Action Representations in Policy Gradient Algorithms

TL;DR

This paper explores how different action representations affect the performance of policy gradient reinforcement learning algorithms, highlighting their influence on learning efficiency and optimization landscape complexity.

Contribution

It provides an analysis framework for understanding the impact of action representations and demonstrates their significant effect on RL performance.

Findings

Action representations significantly influence RL learning performance.

Changes in action representations alter the complexity of the optimization landscape.

Analysis techniques can identify key factors affecting RL efficiency.

Abstract

Reinforcement learning~(RL) is a versatile framework for learning to solve complex real-world tasks. However, influences on the learning performance of RL algorithms are often poorly understood in practice. We discuss different analysis techniques and assess their effectiveness for investigating the impact of action representations in RL. Our experiments demonstrate that the action representation can significantly influence the learning performance on popular RL benchmark tasks. The analysis results indicate that some of the performance differences can be attributed to changes in the complexity of the optimization landscape. Finally, we discuss open challenges of analysis techniques for RL algorithms.