MrCoM: A Meta-Regularized World-Model Generalizing Across Multi-Scenarios

TL;DR

MrCoM introduces a meta-regularized world model that generalizes across multiple scenarios in model-based reinforcement learning, improving prediction accuracy and policy transfer by decomposing state space and regularizing representations.

Contribution

The paper proposes MrCoM, a novel meta-regularized world model that enhances multi-scenario generalization in MBRL through state decomposition and regularization techniques.

Findings

Outperforms previous methods in diverse scenarios

Theoretically bounds generalization error in multi-scenario settings

Demonstrates improved sample efficiency and prediction accuracy

Abstract

Model-based reinforcement learning (MBRL) is a crucial approach to enhance the generalization capabilities and improve the sample efficiency of RL algorithms. However, current MBRL methods focus primarily on building world models for single tasks and rarely address generalization across different scenarios. Building on the insight that dynamics within the same simulation engine share inherent properties, we attempt to construct a unified world model capable of generalizing across different scenarios, named Meta-Regularized Contextual World-Model (MrCoM). This method first decomposes the latent state space into various components based on the dynamic characteristics, thereby enhancing the accuracy of world-model prediction. Further, MrCoM adopts meta-state regularization to extract unified representation of scenario-relevant information, and meta-value regularization to align world-model optimization with policy learning across diverse scenario objectives. We theoretically analyze the generalization error upper bound of MrCoM in multi-scenario settings. We systematically evaluate our algorithm's generalization ability across diverse scenarios, demonstrating significantly better performance than previous state-of-the-art methods.