The Effect of Q-function Reuse on the Total Regret of Tabular, Model-Free, Reinforcement Learning

TL;DR

This paper provides the first theoretical analysis of Q-function reuse in tabular, model-free reinforcement learning, demonstrating it can lead to regret bounds independent of state or action space size, supported by empirical evidence.

Contribution

It offers the first theoretical regret bounds for Q-function reuse in tabular, model-free RL, showing potential for significant sample complexity reduction.

Findings

Regret bound independent of state and action space size

Empirical results support theoretical insights

Q-function reuse improves learning efficiency

Abstract

Some reinforcement learning methods suffer from high sample complexity causing them to not be practical in real-world situations. $Q$-function reuse, a transfer learning method, is one way to reduce the sample complexity of learning, potentially improving usefulness of existing algorithms. Prior work has shown the empirical effectiveness of $Q$-function reuse for various environments when applied to model-free algorithms. To the best of our knowledge, there has been no theoretical work showing the regret of $Q$-function reuse when applied to the tabular, model-free setting. We aim to bridge the gap between theoretical and empirical work in $Q$-function reuse by providing some theoretical insights on the effectiveness of $Q$-function reuse when applied to the $Q$-learning with UCB-Hoeffding algorithm. Our main contribution is showing that in a specific case if $Q$-function reuse is applied to the $Q$-learning with UCB-Hoeffding algorithm it has a regret that is independent of the state or action space. We also provide empirical results supporting our theoretical findings.