Learning to Bridge the Gap: Efficient Novelty Recovery with Planning and Reinforcement Learning

TL;DR

This paper introduces a reinforcement learning-based bridge policy that enables autonomous agents to adapt to environmental novelties by effectively combining learned policies with planning, improving long-horizon decision-making in unpredictable settings.

Contribution

The work proposes a novel RL formulation with a CallPlanner action that allows agents to learn when to switch control back to a planner, enhancing adaptability to environmental changes.

Findings

The approach outperforms baselines in adapting to novelties across multiple simulated domains.

The learned bridge policy generalizes to complex tasks with multiple novelties.

Rapid learning is achieved by leveraging the planner to avoid long-horizon exploration challenges.

Abstract

The real world is unpredictable. Therefore, to solve long-horizon decision-making problems with autonomous robots, we must construct agents that are capable of adapting to changes in the environment during deployment. Model-based planning approaches can enable robots to solve complex, long-horizon tasks in a variety of environments. However, such approaches tend to be brittle when deployed into an environment featuring a novel situation that their underlying model does not account for. In this work, we propose to learn a ``bridge policy'' via Reinforcement Learning (RL) to adapt to such novelties. We introduce a simple formulation for such learning, where the RL problem is constructed with a special ``CallPlanner'' action that terminates the bridge policy and hands control of the agent back to the planner. This allows the RL policy to learn the set of states in which querying the planner and following the returned plan will achieve the goal. We show that this formulation enables the agent to rapidly learn by leveraging the planner's knowledge to avoid challenging long-horizon exploration caused by sparse reward. In experiments across three different simulated domains of varying complexity, we demonstrate that our approach is able to learn policies that adapt to novelty more efficiently than several baselines, including a pure RL baseline. We also demonstrate that the learned bridge policy is generalizable in that it can be combined with the planner to enable the agent to solve more complex tasks with multiple instances of the encountered novelty.