Can Q-Learning be Improved with Advice?

TL;DR

This paper investigates how incorporating predictions about the optimal Q-value function can improve regret bounds in reinforcement learning, especially when predictions satisfy a weak distillation condition, extending algorithms with predictions to complex RL settings.

Contribution

It introduces a method to leverage weakly accurate Q-value predictions to improve regret bounds in RL, and develops an algorithm that performs well even with arbitrary predictions.

Findings

Improved regret bounds when predictions satisfy the distillation condition.

The algorithm achieves sublinear regret with arbitrary predictions.

Extension of prediction-based algorithms from simple online problems to RL.

Abstract

Despite rapid progress in theoretical reinforcement learning (RL) over the last few years, most of the known guarantees are worst-case in nature, failing to take advantage of structure that may be known a priori about a given RL problem at hand. In this paper we address the question of whether worst-case lower bounds for regret in online learning of Markov decision processes (MDPs) can be circumvented when information about the MDP, in the form of predictions about its optimal $Q$-value function, is given to the algorithm. We show that when the predictions about the optimal $Q$-value function satisfy a reasonably weak condition we call distillation, then we can improve regret bounds by replacing the set of state-action pairs with the set of state-action pairs on which the predictions are grossly inaccurate. This improvement holds for both uniform regret bounds and gap-based ones. Further, we are able to achieve this property with an algorithm that achieves sublinear regret when given arbitrary predictions (i.e., even those which are not a distillation). Our work extends a recent line of work on algorithms with predictions, which has typically focused on simple online problems such as caching and scheduling, to the more complex and general problem of reinforcement learning.