The Value of Information When Deciding What to Learn

TL;DR

This paper explores how to optimally select learning targets and acquire information efficiently in sequential decision-making, improving upon existing methods by coupling target design with information acquisition.

Contribution

It introduces a novel approach that combines optimal information acquisition with the design of learning targets, building on information-directed sampling and rate-distortion theory.

Findings

Empirical results demonstrate the value of information in learning target selection.

The proposed method improves efficiency in information acquisition.

Insights connect rate-distortion theory with learning target design.

Abstract

All sequential decision-making agents explore so as to acquire knowledge about a particular target. It is often the responsibility of the agent designer to construct this target which, in rich and complex environments, constitutes a onerous burden; without full knowledge of the environment itself, a designer may forge a sub-optimal learning target that poorly balances the amount of information an agent must acquire to identify the target against the target's associated performance shortfall. While recent work has developed a connection between learning targets and rate-distortion theory to address this challenge and empower agents that decide what to learn in an automated fashion, the proposed algorithm does not optimally tackle the equally important challenge of efficient information acquisition. In this work, building upon the seminal design principle of information-directed sampling (Russo & Van Roy, 2014), we address this shortcoming directly to couple optimal information acquisition with the optimal design of learning targets. Along the way, we offer new insights into learning targets from the literature on rate-distortion theory before turning to empirical results that confirm the value of information when deciding what to learn.