Efficient Model-free Reinforcement Learning in Metric Spaces

TL;DR

This paper introduces a sample-efficient, model-free Q-learning algorithm tailored for continuous state-action spaces with a metric, eliminating the need for a planning oracle and extending discrete RL methods to continuous domains.

Contribution

It presents the first model-free Q-learning algorithm that is efficient in metric spaces with continuous state-action spaces, removing reliance on a planning oracle.

Findings

Achieves sample efficiency in continuous spaces

Extends discrete RL algorithms to continuous domains

Does not require a planning oracle

Abstract

Model-free Reinforcement Learning (RL) algorithms such as Q-learning [Watkins, Dayan 92] have been widely used in practice and can achieve human level performance in applications such as video games [Mnih et al. 15]. Recently, equipped with the idea of optimism in the face of uncertainty, Q-learning algorithms [Jin, Allen-Zhu, Bubeck, Jordan 18] can be proven to be sample efficient for discrete tabular Markov Decision Processes (MDPs) which have finite number of states and actions. In this work, we present an efficient model-free Q-learning based algorithm in MDPs with a natural metric on the state-action space--hence extending efficient model-free Q-learning algorithms to continuous state-action space. Compared to previous model-based RL algorithms for metric spaces [Kakade, Kearns, Langford 03], our algorithm does not require access to a black-box planning oracle.