Learning Off-Policy with Online Planning
Harshit Sikchi, Wenxuan Zhou, David Held

TL;DR
This paper introduces LOOP, a novel off-policy reinforcement learning framework that combines learned models and value functions with online planning, improving performance and safety in robotic tasks.
Contribution
The paper proposes LOOP, a new off-policy RL method integrating learned models and value functions with online planning, along with ARC to address actor divergence, and demonstrates its effectiveness in robotics.
Findings
LOOP outperforms existing methods in robotic tasks.
ARC improves stability by addressing actor divergence.
LOOP effectively incorporates safety constraints.
Abstract
Reinforcement learning (RL) in low-data and risk-sensitive domains requires performant and flexible deployment policies that can readily incorporate constraints during deployment. One such class of policies are the semi-parametric H-step lookahead policies, which select actions using trajectory optimization over a dynamics model for a fixed horizon with a terminal value function. In this work, we investigate a novel instantiation of H-step lookahead with a learned model and a terminal value function learned by a model-free off-policy algorithm, named Learning Off-Policy with Online Planning (LOOP). We provide a theoretical analysis of this method, suggesting a tradeoff between model errors and value function errors and empirically demonstrate this tradeoff to be beneficial in deep reinforcement learning. Furthermore, we identify the "Actor Divergence" issue in this framework and propose…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsReinforcement Learning in Robotics · Software Testing and Debugging Techniques · Software Reliability and Analysis Research
