On- and Off-Policy Monotonic Policy Improvement
Ryo Iwaki, Minoru Asada

TL;DR
This paper introduces a method that guarantees monotonic policy improvement in reinforcement learning by leveraging on- and off-policy samples, supported by a new theoretical bound and a practical trust region algorithm tested on benchmark problems.
Contribution
It provides a novel theoretical bound ensuring monotonic policy improvement from mixed on- and off-policy data, and develops a practical trust region method based on this bound.
Findings
The proposed method guarantees monotonic policy improvement.
The trust region policy optimization with experience replay performs well on benchmarks.
The theoretical bound effectively guides off-policy policy updates.
Abstract
Monotonic policy improvement and off-policy learning are two main desirable properties for reinforcement learning algorithms. In this paper, by lower bounding the performance difference of two policies, we show that the monotonic policy improvement is guaranteed from on- and off-policy mixture samples. An optimization procedure which applies the proposed bound can be regarded as an off-policy natural policy gradient method. In order to support the theoretical result, we provide a trust region policy optimization method using experience replay as a naive application of our bound, and evaluate its performance in two classical benchmark problems.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsReinforcement Learning in Robotics · Optimization and Search Problems · Smart Grid Energy Management
