PrivilegedDreamer: Explicit Imagination of Privileged Information for Rapid Adaptation of Learned Policies

TL;DR

PrivilegedDreamer is a model-based reinforcement learning framework that explicitly estimates hidden parameters in decision problems, enabling rapid adaptation and outperforming existing methods across diverse tasks.

Contribution

It introduces a novel dual recurrent architecture for explicit hidden parameter estimation and conditioning in model-based RL, improving adaptation in HIP-MDPs.

Findings

Outperforms state-of-the-art algorithms on five HIP-MDP tasks

Effective hidden parameter estimation from limited data

Ablation studies validate architecture components

Abstract

Numerous real-world control problems involve dynamics and objectives affected by unobservable hidden parameters, ranging from autonomous driving to robotic manipulation, which cause performance degradation during sim-to-real transfer. To represent these kinds of domains, we adopt hidden-parameter Markov decision processes (HIP-MDPs), which model sequential decision problems where hidden variables parameterize transition and reward functions. Existing approaches, such as domain randomization, domain adaptation, and meta-learning, simply treat the effect of hidden parameters as additional variance and often struggle to effectively handle HIP-MDP problems, especially when the rewards are parameterized by hidden variables. We introduce Privileged-Dreamer, a model-based reinforcement learning framework that extends the existing model-based approach by incorporating an explicit parameter estimation module. PrivilegedDreamer features its novel dual recurrent architecture that explicitly estimates hidden parameters from limited historical data and enables us to condition the model, actor, and critic networks on these estimated parameters. Our empirical analysis on five diverse HIP-MDP tasks demonstrates that PrivilegedDreamer outperforms state-of-the-art model-based, model-free, and domain adaptation learning algorithms. Additionally, we conduct ablation studies to justify the inclusion of each component in the proposed architecture.