Prototypical context-aware dynamics generalization for high-dimensional model-based reinforcement learning

TL;DR

This paper introduces ProtoCAD, a novel model that enhances high-dimensional model-based reinforcement learning by capturing environment context through prototypes, significantly improving dynamics generalization across diverse tasks.

Contribution

ProtoCAD is the first to incorporate temporally consistent prototypes and combined context representations for better dynamics generalization in high-dimensional control tasks.

Findings

ProtoCAD outperforms existing methods in dynamics generalization.

It achieves 13.2% and 26.7% better mean and median performance than RSSM.

Extensive experiments validate its superior generalization ability.

Abstract

The latent world model provides a promising way to learn policies in a compact latent space for tasks with high-dimensional observations, however, its generalization across diverse environments with unseen dynamics remains challenging. Although the recurrent structure utilized in current advances helps to capture local dynamics, modeling only state transitions without an explicit understanding of environmental context limits the generalization ability of the dynamics model. To address this issue, we propose a Prototypical Context-Aware Dynamics (ProtoCAD) model, which captures the local dynamics by time consistent latent context and enables dynamics generalization in high-dimensional control tasks. ProtoCAD extracts useful contextual information with the help of the prototypes clustered over batch and benefits model-based RL in two folds: 1) It utilizes a temporally consistent prototypical regularizer that encourages the prototype assignments produced for different time parts of the same latent trajectory to be temporally consistent instead of comparing the features; 2) A context representation is designed which combines both the projection embedding of latent states and aggregated prototypes and can significantly improve the dynamics generalization ability. Extensive experiments show that ProtoCAD surpasses existing methods in terms of dynamics generalization. Compared with the recurrent-based model RSSM, ProtoCAD delivers 13.2% and 26.7% better mean and median performance across all dynamics generalization tasks.