Biased Dreams: Limitations to Epistemic Uncertainty Quantification in Latent Space Models

TL;DR

This paper reveals that latent space models in reinforcement learning exhibit attractor biases that undermine accurate uncertainty estimation and reward prediction, challenging their reliability.

Contribution

It empirically demonstrates the bias in latent transitions and its impact on uncertainty quantification in latent dynamics models used in reinforcement learning.

Findings

Latent transitions are biased toward well-represented regions.

Biases can cause latent models to deviate from true environment dynamics.

Overestimation of rewards occurs in high-reward attractor regions.

Abstract

Model-Based Reinforcement Learning distinguishes between physical dynamics models operating on proprioceptive inputs and latent dynamics models operating on high-dimensional image observations. A prominent latent approach is the Recurrent State Space Model used in the Dreamer family. While epistemic uncertainty quantification to inform exploration and mitigate model exploitation is well established for physical dynamics models, its transfer to latent dynamics models has received limited scrutiny. We empirically demonstrate that latent transitions are biased toward well-represented regions of latent space, exhibiting an attractor behavior that can deviate from true environment dynamics. As a result, discrepancies in environment dynamics may not manifest in latent space, undermining the reliability of epistemic uncertainty estimates. Because these attractors often lie in high-reward regions, latent rollouts systematically overestimate predicted rewards. Our findings highlight key limitations of epistemic uncertainty estimation in latent dynamics models and motivate more critical evaluation of this method.