M3PO: Massively Multi-Task Model-Based Policy Optimization

TL;DR

M3PO is a scalable multi-task reinforcement learning framework that combines implicit world models with hybrid exploration to improve sample efficiency and generalization, achieving state-of-the-art results.

Contribution

It introduces a novel multi-task model-based policy optimization method that integrates implicit world models with hybrid exploration strategies, addressing limitations of prior approaches.

Findings

Achieves state-of-the-art performance on multiple benchmarks.

Effectively balances exploration and exploitation in multi-task settings.

Demonstrates improved sample efficiency over existing methods.

Abstract

We introduce Massively Multi-Task Model-Based Policy Optimization (M3PO), a scalable model-based reinforcement learning (MBRL) framework designed to address sample inefficiency in single-task settings and poor generalization in multi-task domains. Existing model-based approaches like DreamerV3 rely on pixel-level generative models that neglect control-centric representations, while model-free methods such as PPO suffer from high sample complexity and weak exploration. M3PO integrates an implicit world model, trained to predict task outcomes without observation reconstruction, with a hybrid exploration strategy that combines model-based planning and model-free uncertainty-driven bonuses. This eliminates the bias-variance trade-off in prior methods by using discrepancies between model-based and model-free value estimates to guide exploration, while maintaining stable policy updates through a trust-region optimizer. M3PO provides an efficient and robust alternative to existing model-based policy optimization approaches and achieves state-of-the-art performance across multiple benchmarks.