Assessing Policy, Loss and Planning Combinations in Reinforcement Learning using a New Modular Architecture

TL;DR

This paper introduces a modular software architecture for model-based reinforcement learning, enabling flexible combination of planning algorithms, policies, and loss functions, demonstrated across three diverse environments.

Contribution

A novel modular architecture and set of reusable building blocks for constructing and testing model-based reinforcement learning agents.

Findings

The architecture facilitates problem-specific customization.

The averaged minimax planning algorithm performs well across environments.

Optimal component combinations are highly environment-dependent.

Abstract

The model-based reinforcement learning paradigm, which uses planning algorithms and neural network models, has recently achieved unprecedented results in diverse applications, leading to what is now known as deep reinforcement learning. These agents are quite complex and involve multiple components, factors that can create challenges for research. In this work, we propose a new modular software architecture suited for these types of agents, and a set of building blocks that can be easily reused and assembled to construct new model-based reinforcement learning agents. These building blocks include planning algorithms, policies, and loss functions. We illustrate the use of this architecture by combining several of these building blocks to implement and test agents that are optimized to three different test environments: Cartpole, Minigrid, and Tictactoe. One particular planning algorithm, made available in our implementation and not previously used in reinforcement learning, which we called averaged minimax, achieved good results in the three tested environments. Experiments performed with this architecture have shown that the best combination of planning algorithm, policy, and loss function is heavily problem dependent. This result provides evidence that the proposed architecture, which is modular and reusable, is useful for reinforcement learning researchers who want to study new environments and techniques.