Curricula for Learning Robust Policies with Factored State Representations in Changing Environments

TL;DR

This paper investigates how different curricula in reinforcement learning, utilizing factored state representations, can improve the robustness of policies in dynamic and unpredictable environments, with practical experimental insights.

Contribution

It introduces simple curricula strategies that enhance policy robustness in factored state RL, demonstrating their effectiveness through experiments.

Findings

Curricula varying the variable of highest regret improve robustness.

Factored representations aid in generalization and sample efficiency.

Simple curriculum adjustments significantly enhance policy robustness.

Abstract

Robust policies enable reinforcement learning agents to effectively adapt to and operate in unpredictable, dynamic, and ever-changing real-world environments. Factored representations, which break down complex state and action spaces into distinct components, can improve generalization and sample efficiency in policy learning. In this paper, we explore how the curriculum of an agent using a factored state representation affects the robustness of the learned policy. We experimentally demonstrate three simple curricula, such as varying only the variable of highest regret between episodes, that can significantly enhance policy robustness, offering practical insights for reinforcement learning in complex environments.