Model Based Planning with Energy Based Models
Yilun Du, Toru Lin, Igor Mordatch

TL;DR
This paper demonstrates that energy-based models (EBMs) are effective for model-based planning in reinforcement learning, enabling better online learning, diverse planning, and improved generalization to unseen environments.
Contribution
The paper introduces an online training algorithm for EBMs in RL, showing their advantages over feed-forward models for planning, inference, and exploration.
Findings
EBMs support better online learning than feed-forward networks.
EBMs enable diverse and generalizable state space plans.
Online EBM training enhances state exploration strategies.
Abstract
Model-based planning holds great promise for improving both sample efficiency and generalization in reinforcement learning (RL). We show that energy-based models (EBMs) are a promising class of models to use for model-based planning. EBMs naturally support inference of intermediate states given start and goal state distributions. We provide an online algorithm to train EBMs while interacting with the environment, and show that EBMs allow for significantly better online learning than corresponding feed-forward networks. We further show that EBMs support maximum entropy state inference and are able to generate diverse state space plans. We show that inference purely in state space - without planning actions - allows for better generalization to previously unseen obstacles in the environment and prevents the planner from exploiting the dynamics model by applying uncharacteristic action…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsReinforcement Learning in Robotics · Robotic Path Planning Algorithms · Artificial Intelligence in Games
Methodsenergy-based model
