Direct Uncertainty Estimation in Reinforcement Learning

TL;DR

This paper explores a direct method for estimating uncertainty in reinforcement learning, aiming to improve exploration strategies by measuring uncertainty of the action-value function without extensive model propagation.

Contribution

It introduces the concept of directly measuring uncertainty in the action-value function and analyzes its sufficiency as an alternative to traditional probabilistic methods.

Findings

Direct uncertainty measurement can potentially simplify exploration in RL.

Propagation of uncertainty via Bellman iterations is computationally demanding.

The analysis suggests feasible approaches for direct uncertainty estimation.

Abstract

Optimal probabilistic approach in reinforcement learning is computationally infeasible. Its simplification consisting in neglecting difference between true environment and its model estimated using limited number of observations causes exploration vs exploitation problem. Uncertainty can be expressed in terms of a probability distribution over the space of environment models, and this uncertainty can be propagated to the action-value function via Bellman iterations, which are computationally insufficiently efficient though. We consider possibility of directly measuring uncertainty of the action-value function, and analyze sufficiency of this facilitated approach.