$\lambda$-models: Effective Decision-Aware Reinforcement Learning with Latent Models

TL;DR

This paper investigates decision-aware reinforcement learning with latent models, emphasizing the importance of design choices like latent models and analyzing the MuZero loss function's bias in stochastic environments, providing practical insights.

Contribution

It offers a comprehensive theoretical and empirical analysis of decision-aware RL models, highlighting the critical role of latent models and loss function biases in continuous control tasks.

Findings

Latent models are crucial for high performance in decision-aware RL.

MuZero's loss function is biased in stochastic environments, affecting performance.

Design choices from MuZero are validated as vital for effective algorithms.

Abstract

The idea of decision-aware model learning, that models should be accurate where it matters for decision-making, has gained prominence in model-based reinforcement learning. While promising theoretical results have been established, the empirical performance of algorithms leveraging a decision-aware loss has been lacking, especially in continuous control problems. In this paper, we present a study on the necessary components for decision-aware reinforcement learning models and we showcase design choices that enable well-performing algorithms. To this end, we provide a theoretical and empirical investigation into algorithmic ideas in the field. We highlight that empirical design decisions established in the MuZero line of works, most importantly the use of a latent model, are vital to achieving good performance for related algorithms. Furthermore, we show that the MuZero loss function is biased in stochastic environments and establish that this bias has practical consequences. Building on these findings, we present an overview of which decision-aware loss functions are best used in what empirical scenarios, providing actionable insights to practitioners in the field.