Operator Splitting Value Iteration
Amin Rakhsha, Andrew Wang, Mohammad Ghavamzadeh, Amir-massoud, Farahmand

TL;DR
This paper presents Operator Splitting Value Iteration (OS-VI) and OS-Dyna, innovative algorithms that leverage approximate models to significantly speed up convergence in planning and reinforcement learning for discounted MDPs.
Contribution
It introduces a novel operator splitting approach for value iteration and a sample-based variant that maintains convergence despite model inaccuracies.
Findings
OS-VI converges faster with accurate models.
OS-Dyna maintains convergence with approximate models.
The algorithms outperform traditional methods in convergence speed.
Abstract
We introduce new planning and reinforcement learning algorithms for discounted MDPs that utilize an approximate model of the environment to accelerate the convergence of the value function. Inspired by the splitting approach in numerical linear algebra, we introduce Operator Splitting Value Iteration (OS-VI) for both Policy Evaluation and Control problems. OS-VI achieves a much faster convergence rate when the model is accurate enough. We also introduce a sample-based version of the algorithm called OS-Dyna. Unlike the traditional Dyna architecture, OS-Dyna still converges to the correct value function in presence of model approximation error.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
Taxonomy
TopicsReinforcement Learning in Robotics · Auction Theory and Applications · Formal Methods in Verification
