Deterministic Sequencing of Exploration and Exploitation for Reinforcement Learning

TL;DR

This paper introduces DSEE, a deterministic algorithm for reinforcement learning that interleaves exploration and exploitation to efficiently learn system models and policies with sub-linear regret growth.

Contribution

The paper presents a novel deterministic sequencing approach for exploration and exploitation in model-based RL, improving learning efficiency and regret bounds.

Findings

DSEE achieves sub-linear cumulative regret over time.

The algorithm effectively balances exploration and exploitation epochs.

Theoretical analysis confirms the robustness of the learned policy.

Abstract

We propose Deterministic Sequencing of Exploration and Exploitation (DSEE) algorithm with interleaving exploration and exploitation epochs for model-based RL problems that aim to simultaneously learn the system model, i.e., a Markov decision process (MDP), and the associated optimal policy. During exploration, DSEE explores the environment and updates the estimates for expected reward and transition probabilities. During exploitation, the latest estimates of the expected reward and transition probabilities are used to obtain a robust policy with high probability. We design the lengths of the exploration and exploitation epochs such that the cumulative regret grows as a sub-linear function of time.