Improving planning and MBRL with temporally-extended actions

TL;DR

This paper introduces temporally-extended actions in reinforcement learning to directly control decision timescales, leading to faster planning, improved solutions, and enhanced model-based learning efficiency.

Contribution

It proposes controlling action durations as an optimization variable, integrating multi-armed bandits for automatic duration selection, and demonstrating significant improvements in planning and MBRL.

Findings

Faster planning and better solutions in various tasks.

Reduced model errors and training time in MBRL.

Ability to solve previously unsolvable problems.

Abstract

Continuous time systems are often modeled using discrete time dynamics but this requires a small simulation step to maintain accuracy. In turn, this requires a large planning horizon which leads to computationally demanding planning problems and reduced performance. Previous work in model-free reinforcement learning has partially addressed this issue using action repeats where a policy is learned to determine a discrete action duration. Instead we propose to control the continuous decision timescale directly by using temporally-extended actions and letting the planner treat the duration of the action as an additional optimization variable along with the standard action variables. This additional structure has multiple advantages. It speeds up simulation time of trajectories and, importantly, it allows for deep horizon search in terms of primitive actions while using a shallow search depth in the planner. In addition, in the model-based reinforcement learning (MBRL) setting, it reduces compounding errors from model learning and improves training time for models. We show that this idea is effective and that the range for action durations can be automatically selected using a multi-armed bandit formulation and integrated into the MBRL framework. An extensive experimental evaluation both in planning and in MBRL, shows that our approach yields faster planning, better solutions, and that it enables solutions to problems that are not solved in the standard formulation.