Acting upon Imagination: when to trust imagined trajectories in model based reinforcement learning

TL;DR

This paper introduces uncertainty estimation techniques for model-based reinforcement learning to determine when imagined trajectories can be trusted, reducing unnecessary re-planning and computational costs while maintaining performance.

Contribution

It proposes novel uncertainty estimation methods for online evaluation of imagined trajectories in MBRL, improving efficiency without sacrificing accuracy.

Findings

Significant reduction in computational costs.

Effective avoidance of unnecessary trajectory re-planning.

Maintained performance with fewer re-plans.

Abstract

Model-based reinforcement learning (MBRL) aims to learn model(s) of the environment dynamics that can predict the outcome of its actions. Forward application of the model yields so called imagined trajectories (sequences of action, predicted state-reward) used to optimize the set of candidate actions that maximize expected reward. The outcome, an ideal imagined trajectory or plan, is imperfect and typically MBRL relies on model predictive control (MPC) to overcome this by continuously re-planning from scratch, incurring thus major computational cost and increasing complexity in tasks with longer receding horizon. We propose uncertainty estimation methods for online evaluation of imagined trajectories to assess whether further planned actions can be trusted to deliver acceptable reward. These methods include comparing the error after performing the last action with the standard expected error and using model uncertainty to assess the deviation from expected outcomes. Additionally, we introduce methods that exploit the forward propagation of the dynamics model to evaluate if the remainder of the plan aligns with expected results and assess the remainder of the plan in terms of the expected reward. Our experiments demonstrate the effectiveness of the proposed uncertainty estimation methods by applying them to avoid unnecessary trajectory replanning in a shooting MBRL setting. Results highlight significant reduction on computational costs without sacrificing performance.