Coarse-Grained Smoothness for RL in Metric Spaces

TL;DR

This paper introduces a new coarse-grained smoothness concept for Q-functions in reinforcement learning, addressing limitations of Lipschitz continuity and enabling tighter bounds for better learning in continuous spaces.

Contribution

It proposes a generalized smoothness definition that is more applicable than Lipschitz continuity, with theoretical analysis and implications for control and exploration.

Findings

New smoothness definition generalizes Lipschitz continuity

Tighter bounds on Q-functions improve learning efficiency

Enhanced control and exploration in continuous domains

Abstract

Principled decision-making in continuous state--action spaces is impossible without some assumptions. A common approach is to assume Lipschitz continuity of the Q-function. We show that, unfortunately, this property fails to hold in many typical domains. We propose a new coarse-grained smoothness definition that generalizes the notion of Lipschitz continuity, is more widely applicable, and allows us to compute significantly tighter bounds on Q-functions, leading to improved learning. We provide a theoretical analysis of our new smoothness definition, and discuss its implications and impact on control and exploration in continuous domains.