Improved Model-based Reinforcement Learning with Smooth Kernels

TL;DR

This paper introduces a kernel-smoothing model-based reinforcement learning method that leverages MDP smoothness and Bernstein bonuses to improve regret bounds in finite-horizon settings.

Contribution

It develops a novel kernel-smoothing approach with Bernstein-style exploration bonuses, achieving improved regret bounds over existing methods.

Findings

Achieves better regret bounds with respect to the horizon.

Introduces a new Bernstein-type concentration inequality for martingales.

Demonstrates the effectiveness of smooth kernel methods in RL.

Abstract

For continuous state-action space scenarios, classical reinforcement learning (RL) theory predominantly focuses on low-rank Markov decision processes (MDPs), which provide sample-efficient guarantees at the expense of restrictive structural assumptions. Kernel smoothing model-based approaches offer a promising alternative paradigm that instead leverages the smoothness of the MDP and employs non-parametric kernel smoothing estimates of transition dynamics. This paper proposes a new kernel-smoothing model-based approach for online reinforcement learning in finite-horizon settings under Lipschitz continuity assumptions on the MDP. By incorporating a Bernstein-style exploration bonus into the kernel smoothing framework, our method achieves a regret bound which improves upon the state-of-the-art regret bound in its dependence on the horizon. The theoretical advancement relies on a delicate analysis of the synergy between Bernstein-style bonuses and kernel smoothing, where a new tight Bernstein-type concentration inequality for martingales may be of independent interest.