# Robust Exploration with Tight Bayesian Plausibility Sets

## TL;DR

The paper introduces OFVF, a Bayesian exploration algorithm for reinforcement learning that constructs tight plausibility sets based on value function structure, leading to robust and efficient exploration.

## Contribution

It presents a novel Bayesian method for constructing plausibility sets that are tighter than confidence intervals, improving exploration in reinforcement learning.

## Key findings

- OFVF achieves lower exploration costs in experiments.
- Theoretical analysis confirms robustness of OFVF.
- Method leverages prior information for improved performance.

## Abstract

Optimism about the poorly understood states and actions is the main driving force of exploration for many provably-efficient reinforcement learning algorithms. We propose optimism in the face of sensible value functions (OFVF)- a novel data-driven Bayesian algorithm to constructing Plausibility sets for MDPs to explore robustly minimizing the worst case exploration cost. The method computes policies with tighter optimistic estimates for exploration by introducing two new ideas. First, it is based on Bayesian posterior distributions rather than distribution-free bounds. Second, OFVF does not construct plausibility sets as simple confidence intervals. Confidence intervals as plausibility sets are a sufficient but not a necessary condition. OFVF uses the structure of the value function to optimize the location and shape of the plausibility set to guarantee upper bounds directly without necessarily enforcing the requirement for the set to be a confidence interval. OFVF proceeds in an episodic manner, where the duration of the episode is fixed and known. Our algorithm is inherently Bayesian and can leverage prior information. Our theoretical analysis shows the robustness of OFVF, and the empirical results demonstrate its practical promise.