Reinforcement Learning with Multi-Step Lookahead Information Via Adaptive Batching

TL;DR

This paper introduces adaptive batching policies for tabular reinforcement learning with multi-step lookahead, providing a new approach that improves over fixed batching and model predictive control by deriving optimal Bellman equations and an optimistic learning algorithm.

Contribution

It proposes adaptive batching policies for multi-step lookahead RL, deriving Bellman equations and an optimistic algorithm with near-optimal regret bounds.

Findings

Adaptive batching policies outperform fixed batching in RL tasks.

The proposed algorithm achieves order-optimal regret bounds.

Lookahead horizon impacts regret bounds only by a small constant.

Abstract

We study tabular reinforcement learning problems with multiple steps of lookahead information. Before acting, the learner observes $\ell$ steps of future transition and reward realizations: the exact state the agent would reach and the rewards it would collect under any possible course of action. While it has been shown that such information can drastically boost the value, finding the optimal policy is NP-hard, and it is common to apply one of two tractable heuristics: processing the lookahead in chunks of predefined sizes ('fixed batching policies'), and model predictive control. We first illustrate the problems with these two approaches and propose utilizing the lookahead in adaptive (state-dependent) batches; we refer to such policies as adaptive batching policies (ABPs). We derive the optimal Bellman equations for these strategies and design an optimistic regret-minimizing algorithm that enables learning the optimal ABP when interacting with unknown environments. Our regret bounds are order-optimal up to a potential factor of the lookahead horizon $\ell$, which can usually be considered a small constant.