AsyncQVI: Asynchronous-Parallel Q-Value Iteration for Discounted Markov Decision Processes with Near-Optimal Sample Complexity

TL;DR

AsyncQVI is an asynchronous-parallel algorithm for discounted Markov decision processes that achieves near-optimal sample complexity with low memory usage, making it suitable for large-scale problems and demonstrating high efficiency and linear speedup.

Contribution

It introduces AsyncQVI, the first asynchronous-parallel algorithm with near-optimal sample complexity for discounted MDPs, combining low memory footprint and high efficiency.

Findings

Achieves near-optimal sample complexity matching theoretical lower bounds.

Demonstrates high efficiency and linear parallel speedup in numerical tests.

Uses memory of size proportional to the number of states, suitable for large-scale applications.

Abstract

In this paper, we propose AsyncQVI, an asynchronous-parallel Q-value iteration for discounted Markov decision processes whose transition and reward can only be sampled through a generative model. Given such a problem with $|\mathcal{S}|$ states, $|\mathcal{A}|$ actions, and a discounted factor $\gamma\in(0,1)$, AsyncQVI uses memory of size $\mathcal{O}(|\mathcal{S}|)$ and returns an $\varepsilon$-optimal policy with probability at least $1-\delta$ using $$\tilde{\mathcal{O}}\big(\frac{|\mathcal{S}||\mathcal{A}|}{(1-\gamma)^5\varepsilon^2}\log(\frac{1}{\delta})\big)$$ samples. AsyncQVI is also the first asynchronous-parallel algorithm for discounted Markov decision processes that has a sample complexity, which nearly matches the theoretical lower bound. The relatively low memory footprint and parallel ability make AsyncQVI suitable for large-scale applications. In numerical tests, we compare AsyncQVI with four sample-based value iteration methods. The results show that our algorithm is highly efficient and achieves linear parallel speedup.