Mind Dreamer: Untethering Imagination via Active Latent Intervention on Latent Manifolds

TL;DR

Mind Dreamer introduces Active Latent Intervention to enhance model-based RL by synthesizing plausible non-continuous latent states, improving exploration and sample efficiency in complex environments.

Contribution

It proposes a novel framework that transcends Markovian constraints using adversarial latent jumps and new value functions, advancing latent imagination in RL.

Findings

Achieves 1.67× speedup over DreamerV3 on DeepMind Control Suite.

Reaches 8.8× speedup in sparse-reward tasks.

Theoretically establishes quadratic discount for uncertainty propagation.

Abstract

Model-Based Reinforcement Learning (MBRL) leverages latent imagination for sample efficiency, yet remains constrained by Historical Tethering: imagination is typically initialized from observed states. This creates a learning asymmetry, where the world model's manifold discovery outpaces the policy's sparse-reward optimization. We propose Mind Dreamer (MD), a framework that operationalizes Active Latent Intervention (ALI) to transcend Markovian continuity. MD reformulates discovery as the minimization of a global Relay Manifold Expected Free Energy (R-EFE); by sampling initial states from a learned generator $s_0 \sim p_{gen}(\cdot)$ rather than the historical buffer, MD utilizes an adversarial generator to synthesize non-continuous latent jumps to epistemic blind spots that are physically plausible yet cognitively challenging. To resolve the credit assignment paradox across these spatial ruptures, we derive the Relay Value Function (RVF) and Relay Uncertainty Function (RUF). These potentials treat synthesized anchors as counterfactual intermediary states, propagating pragmatic and epistemic value through a principled Bellman-style formulation. Notably, we prove that uncertainty propagation across discontinuities necessitates a quadratic discount $\gamma^2$, establishing a formal epistemic horizon. Theoretically, MD approximates a variance-minimizing importance sampler that expands the manifold's spectral gap, reducing the hitting time to critical bottleneck states. Empirically, MD achieves a 1.67$\times$ average speedup over DreamerV3 on DeepMind Control Suite, reaching 8.8$\times$ in sparse-reward tasks.