Metrics and continuity in reinforcement learning

TL;DR

This paper introduces a formal framework using metrics to define state similarity topologies in reinforcement learning, analyzing their theoretical impacts and empirically comparing their effects on algorithm performance.

Contribution

It provides a unified formalism for metrics in reinforcement learning and explores their hierarchical relationships and implications.

Findings

Different metrics induce distinct topologies affecting learning performance

Theoretical hierarchy of metrics influences MDP properties

Empirical results show metric choice impacts generalization and efficiency

Abstract

In most practical applications of reinforcement learning, it is untenable to maintain direct estimates for individual states; in continuous-state systems, it is impossible. Instead, researchers often leverage state similarity (whether explicitly or implicitly) to build models that can generalize well from a limited set of samples. The notion of state similarity used, and the neighbourhoods and topologies they induce, is thus of crucial importance, as it will directly affect the performance of the algorithms. Indeed, a number of recent works introduce algorithms assuming the existence of "well-behaved" neighbourhoods, but leave the full specification of such topologies for future work. In this paper we introduce a unified formalism for defining these topologies through the lens of metrics. We establish a hierarchy amongst these metrics and demonstrate their theoretical implications on the Markov Decision Process specifying the reinforcement learning problem. We complement our theoretical results with empirical evaluations showcasing the differences between the metrics considered.