STEERING: Stein Information Directed Exploration for Model-Based Reinforcement Learning

TL;DR

This paper introduces STEIN, a novel exploration method for model-based reinforcement learning that uses kernelized Stein discrepancy to efficiently estimate information gain, achieving better regret bounds and practical performance.

Contribution

The paper proposes a new exploration incentive based on IPM and KSD, along with a novel algorithm STEIN, improving computational efficiency and regret bounds in model-based RL.

Findings

Achieves sublinear Bayesian regret with STEIN.

Outperforms prior methods in experiments.

Computationally affordable exploration strategy.

Abstract

Directed Exploration is a crucial challenge in reinforcement learning (RL), especially when rewards are sparse. Information-directed sampling (IDS), which optimizes the information ratio, seeks to do so by augmenting regret with information gain. However, estimating information gain is computationally intractable or relies on restrictive assumptions which prohibit its use in many practical instances. In this work, we posit an alternative exploration incentive in terms of the integral probability metric (IPM) between a current estimate of the transition model and the unknown optimal, which under suitable conditions, can be computed in closed form with the kernelized Stein discrepancy (KSD). Based on KSD, we develop a novel algorithm \algo: \textbf{STE}in information dir\textbf{E}cted exploration for model-based \textbf{R}einforcement Learn\textbf{ING}. To enable its derivation, we develop fundamentally new variants of KSD for discrete conditional distributions. {We further establish that {\algo} archives sublinear Bayesian regret, improving upon prior learning rates of information-augmented MBRL.} Experimentally, we show that the proposed algorithm is computationally affordable and outperforms several prior approaches.