Opportunistic Episodic Reinforcement Learning

TL;DR

This paper introduces opportunistic reinforcement learning, where exploration and exploitation are dynamically balanced based on an external variation factor, leading to improved regret bounds and performance in episodic MDPs.

Contribution

The paper proposes a novel opportunistic RL framework and develops algorithms that adapt exploration based on environmental variation, with theoretical regret bounds and empirical validation.

Findings

OppUCRL2 achieves an $ ilde{O}(HS \\sqrt{AT})$ regret bound.

Both OppUCRL2 and OppPSRL outperform their original algorithms in simulations.

The framework effectively exploits environmental variation to improve learning efficiency.

Abstract

In this paper, we propose and study opportunistic reinforcement learning - a new variant of reinforcement learning problems where the regret of selecting a suboptimal action varies under an external environmental condition known as the variation factor. When the variation factor is low, so is the regret of selecting a suboptimal action and vice versa. Our intuition is to exploit more when the variation factor is high, and explore more when the variation factor is low. We demonstrate the benefit of this novel framework for finite-horizon episodic MDPs by designing and evaluating OppUCRL2 and OppPSRL algorithms. Our algorithms dynamically balance the exploration-exploitation trade-off for reinforcement learning by introducing variation factor-dependent optimism to guide exploration. We establish an $\tilde{O}(HS \sqrt{AT})$ regret bound for the OppUCRL2 algorithm and show through simulations that both OppUCRL2 and OppPSRL algorithm outperform their original corresponding algorithms.