Sharp Analysis of Smoothed Bellman Error Embedding

TL;DR

This paper provides a theoretical analysis of the SBEED reinforcement learning algorithm, establishing near-optimal performance guarantees in batch settings and improving upon previous bounds related to planning horizon and sample size.

Contribution

It offers the first rigorous performance guarantees for SBEED in batch reinforcement learning, highlighting the impact of function class capacity and distribution shift.

Findings

Proves near-optimal performance bounds for SBEED

Shows dependence of guarantees on function approximation and distribution shift

Improves prior theoretical bounds on sample complexity and horizon dependence

Abstract

The \textit{Smoothed Bellman Error Embedding} algorithm~\citep{dai2018sbeed}, known as SBEED, was proposed as a provably convergent reinforcement learning algorithm with general nonlinear function approximation. It has been successfully implemented with neural networks and achieved strong empirical results. In this work, we study the theoretical behavior of SBEED in batch-mode reinforcement learning. We prove a near-optimal performance guarantee that depends on the representation power of the used function classes and a tight notion of the distribution shift. Our results improve upon prior guarantees for SBEED in ~\citet{dai2018sbeed} in terms of the dependence on the planning horizon and on the sample size. Our analysis builds on the recent work of ~\citet{Xie2020} which studies a related algorithm MSBO, that could be interpreted as a \textit{non-smooth} counterpart of SBEED.