Adaptive Planning with Generative Models under Uncertainty

TL;DR

This paper introduces an adaptive planning method using generative models and predictive uncertainty to reduce replanning frequency in decision-making tasks, maintaining performance while significantly decreasing computational load.

Contribution

It presents a novel adaptive planning policy that leverages generative models' long-horizon predictions and uncertainty estimates to dynamically adjust planning intervals.

Findings

Replanning frequency reduced to 10% of steps

Maintains performance in locomotion tasks

Demonstrates efficiency of generative models for decision-making

Abstract

Planning with generative models has emerged as an effective decision-making paradigm across a wide range of domains, including reinforcement learning and autonomous navigation. While continuous replanning at each timestep might seem intuitive because it allows decisions to be made based on the most recent environmental observations, it results in substantial computational challenges, primarily due to the complexity of the generative model's underlying deep learning architecture. Our work addresses this challenge by introducing a simple adaptive planning policy that leverages the generative model's ability to predict long-horizon state trajectories, enabling the execution of multiple actions consecutively without the need for immediate replanning. We propose to use the predictive uncertainty derived from a Deep Ensemble of inverse dynamics models to dynamically adjust the intervals between planning sessions. In our experiments conducted on locomotion tasks within the OpenAI Gym framework, we demonstrate that our adaptive planning policy allows for a reduction in replanning frequency to only about 10% of the steps without compromising the performance. Our results underscore the potential of generative modeling as an efficient and effective tool for decision-making.