The Efficacy of Pessimism in Asynchronous Q-Learning

TL;DR

This paper introduces a pessimism-based framework for asynchronous Q-learning that improves sample efficiency and adaptivity, especially with partial data coverage, and provides the first theoretical validation for pessimism in Markovian non-i.i.d. data.

Contribution

It develops a novel pessimism-incorporating algorithmic framework for asynchronous Q-learning with theoretical guarantees under partial data coverage.

Findings

Achieves near-optimal sample complexity with variance reduction.

Supports partial state-action space coverage unlike prior theories.

Provides the first theoretical validation of pessimism in Markovian non-i.i.d. data.

Abstract

This paper is concerned with the asynchronous form of Q-learning, which applies a stochastic approximation scheme to Markovian data samples. Motivated by the recent advances in offline reinforcement learning, we develop an algorithmic framework that incorporates the principle of pessimism into asynchronous Q-learning, which penalizes infrequently-visited state-action pairs based on suitable lower confidence bounds (LCBs). This framework leads to, among other things, improved sample efficiency and enhanced adaptivity in the presence of near-expert data. Our approach permits the observed data in some important scenarios to cover only partial state-action space, which is in stark contrast to prior theory that requires uniform coverage of all state-action pairs. When coupled with the idea of variance reduction, asynchronous Q-learning with LCB penalization achieves near-optimal sample complexity, provided that the target accuracy level is small enough. In comparison, prior works were suboptimal in terms of the dependency on the effective horizon even when i.i.d. sampling is permitted. Our results deliver the first theoretical support for the use of pessimism principle in the presence of Markovian non-i.i.d. data.