A Cantor-Kantorovich Metric Between Markov Decision Processes with Application to Transfer Learning

TL;DR

This paper introduces a new metric based on the Cantor-Kantorovich distance for Markov Decision Processes, enabling better transfer learning performance forecasting in reinforcement learning.

Contribution

It extends the Cantor-Kantorovich distance to MDPs, providing a well-defined, efficiently approximable metric with practical applications in transfer learning.

Findings

The metric is well-defined and computationally feasible for finite horizons.

Numerical experiments demonstrate its effectiveness in transfer learning performance prediction.

Potential to improve transfer learning strategies in reinforcement learning.

Abstract

We extend the notion of Cantor-Kantorovich distance between Markov chains introduced by (Banse et al., 2023) in the context of Markov Decision Processes (MDPs). The proposed metric is well-defined and can be efficiently approximated given a finite horizon. Then, we provide numerical evidences that the latter metric can lead to interesting applications in the field of reinforcement learning. In particular, we show that it could be used for forecasting the performance of transfer learning algorithms.