Knowledge is reward: Learning optimal exploration by predictive reward cashing

TL;DR

This paper introduces a novel approach to optimal exploration in reinforcement learning by leveraging the concept of predictive reward cashing, simplifying the Bayes-adaptive exploration problem and enabling effective learning without heuristics.

Contribution

The paper proposes a new mathematical structure for Bayes-adaptive exploration using cross-value, which simplifies the problem and improves learning efficiency.

Findings

Enables learning complex information gathering tasks without shaping or heuristics.

Dramatically simplifies the computational complexity of Bayes-adaptive exploration.

Shows improved performance over standard RL algorithms in experiments.

Abstract

There is a strong link between the general concept of intelligence and the ability to collect and use information. The theory of Bayes-adaptive exploration offers an attractive optimality framework for training machines to perform complex information gathering tasks. However, the computational complexity of the resulting optimal control problem has limited the diffusion of the theory to mainstream deep AI research. In this paper we exploit the inherent mathematical structure of Bayes-adaptive problems in order to dramatically simplify the problem by making the reward structure denser while simultaneously decoupling the learning of exploitation and exploration policies. The key to this simplification comes from the novel concept of cross-value (i.e. the value of being in an environment while acting optimally according to another), which we use to quantify the value of currently available information. This results in a new denser reward structure that "cashes in" all future rewards that can be predicted from the current information state. In a set of experiments we show that the approach makes it possible to learn challenging information gathering tasks without the use of shaping and heuristic bonuses in situations where the standard RL algorithms fail.