Universal Value-Function Uncertainties

TL;DR

This paper introduces UVU, a novel method for estimating value-function uncertainty in reinforcement learning that is computationally efficient and theoretically grounded, performing comparably to ensembles in offline RL tasks.

Contribution

UVU provides a simple, theoretically justified approach to quantify value uncertainty using prediction errors against a fixed network, reducing computational costs compared to ensembles.

Findings

UVU errors are equivalent to ensemble variance in the infinite-width limit.

UVU achieves similar performance to large ensembles in offline RL tasks.

UVU offers a computationally efficient alternative to ensemble methods.

Abstract

Estimating epistemic uncertainty in value functions is a crucial challenge for many aspects of reinforcement learning (RL), including efficient exploration, safe decision-making, and offline RL. While deep ensembles provide a robust method for quantifying value uncertainty, they come with significant computational overhead. Single-model methods, while computationally favorable, often rely on heuristics and typically require additional propagation mechanisms for myopic uncertainty estimates. In this work we introduce universal value-function uncertainties (UVU), which, similar in spirit to random network distillation (RND), quantify uncertainty as squared prediction errors between an online learner and a fixed, randomly initialized target network. Unlike RND, UVU errors reflect policy-conditional value uncertainty, incorporating the future uncertainties any given policy may encounter. This is due to the training procedure employed in UVU: the online network is trained using temporal difference learning with a synthetic reward derived from the fixed, randomly initialized target network. We provide an extensive theoretical analysis of our approach using neural tangent kernel (NTK) theory and show that in the limit of infinite network width, UVU errors are exactly equivalent to the variance of an ensemble of independent universal value functions. Empirically, we show that UVU achieves equal performance to large ensembles on challenging multi-task offline RL settings, while offering simplicity and substantial computational savings.