Improved Regret for Efficient Online Reinforcement Learning with Linear Function Approximation

TL;DR

This paper introduces a new efficient algorithm for online reinforcement learning with linear function approximation under adversarial costs, achieving significantly improved regret bounds in challenging settings.

Contribution

It presents a novel policy optimization algorithm combining mirror descent and least squares evaluation, with improved regret bounds for unknown dynamics and bandit feedback.

Findings

Achieves $ ilde{O}(K^{6/7})$ regret without simulator.

Achieves $ ilde{O}(K^{2/3})$ regret with a simulator.

Outperforms previous state-of-the-art regret bounds.

Abstract

We study reinforcement learning with linear function approximation and adversarially changing cost functions, a setup that has mostly been considered under simplifying assumptions such as full information feedback or exploratory conditions.We present a computationally efficient policy optimization algorithm for the challenging general setting of unknown dynamics and bandit feedback, featuring a combination of mirror-descent and least squares policy evaluation in an auxiliary MDP used to compute exploration bonuses.Our algorithm obtains an $\widetilde O(K^{6/7})$ regret bound, improving significantly over previous state-of-the-art of $\widetilde O (K^{14/15})$ in this setting. In addition, we present a version of the same algorithm under the assumption a simulator of the environment is available to the learner (but otherwise no exploratory assumptions are made), and prove it obtains state-of-the-art regret of $\widetilde O (K^{2/3})$.