On the instability of local learning algorithms: Q-learning can fail in infinite state spaces

TL;DR

This paper reveals fundamental instability issues of Q-learning in infinite state spaces, demonstrating that it can fail to converge or be sub-optimal, especially under certain initial conditions and drifts, with implications for reinforcement learning in complex environments.

Contribution

The paper introduces Local Learning Processes (LLPs) and analyzes their instability, providing new theoretical insights into the limitations of Q-learning in infinite state spaces.

Findings

Q-learning can be unstable in infinite state spaces.

LLPs exhibit instability under certain conditions.

Q-learning may be strictly sub-optimal in infinite settings.

Abstract

We investigate the challenges of applying model-free reinforcement learning algorithms, like online Q-learning, to infinite state space Markov Decision Processes (MDPs). We first introduce the notion of Local Learning Processes (LLPs), where agents make decisions based solely on local information, and we show that Q-learning can be seen as a specific instance of an LLP. Using renewal techniques, we analyze LLPs and demonstrate their instability under certain drift and initial conditions, revealing fundamental limitations in infinite state spaces. In particular, we show that while asymptotically optimal in finite settings, Q-learning can face instability and strict sub-optimality in infinite spaces. Our findings are illustrated through queueing system examples drawn from load balancing and server allocation. The study underscores the need for new theoretical frameworks and suggests future research into nonlocal Q-learning variants.