Unifying Model-Free Efficiency and Model-Based Representations via Latent Dynamics

TL;DR

Unified Latent Dynamics (ULD) is a reinforcement learning method that combines model-free efficiency with model-based representational strengths using latent space embeddings, achieving high performance across diverse domains without planning overhead.

Contribution

The paper introduces ULD, a novel RL algorithm that unifies model-free and model-based approaches through latent space embeddings, with theoretical guarantees and broad empirical success.

Findings

Matches or exceeds specialized baselines in 80 environments

Supports a single hyperparameter set across diverse tasks

Achieves cross-domain competence with minimal tuning

Abstract

We present Unified Latent Dynamics (ULD), a novel reinforcement learning algorithm that unifies the efficiency of model-free methods with the representational strengths of model-based approaches, without incurring planning overhead. By embedding state-action pairs into a latent space in which the true value function is approximately linear, our method supports a single set of hyperparameters across diverse domains -- from continuous control with low-dimensional and pixel inputs to high-dimensional Atari games. We prove that, under mild conditions, the fixed point of our embedding-based temporal-difference updates coincides with that of a corresponding linear model-based value expansion, and we derive explicit error bounds relating embedding fidelity to value approximation quality. In practice, ULD employs synchronized updates of encoder, value, and policy networks, auxiliary losses for short-horizon predictive dynamics, and reward-scale normalization to ensure stable learning under sparse rewards. Evaluated on 80 environments spanning Gym locomotion, DeepMind Control (proprioceptive and visual), and Atari, our approach matches or exceeds the performance of specialized model-free and general model-based baselines -- achieving cross-domain competence with minimal tuning and a fraction of the parameter footprint. These results indicate that value-aligned latent representations alone can deliver the adaptability and sample efficiency traditionally attributed to full model-based planning.