Mastering Massive Multi-Task Reinforcement Learning via Mixture-of-Expert Decision Transformer

TL;DR

This paper introduces M3DT, a mixture-of-experts framework for massive multi-task reinforcement learning, which significantly improves scalability and performance across a large number of tasks by enhancing the Decision Transformer architecture.

Contribution

We propose M3DT, a novel MoE-based architecture with a three-stage training process that enables efficient scaling to hundreds of tasks in offline reinforcement learning.

Findings

M3DT outperforms existing methods on large-scale multi-task benchmarks.

Increasing the number of experts improves model performance and task scalability.

M3DT successfully extends to 160 tasks with superior results.

Abstract

Despite recent advancements in offline multi-task reinforcement learning (MTRL) have harnessed the powerful capabilities of the Transformer architecture, most approaches focus on a limited number of tasks, with scaling to extremely massive tasks remaining a formidable challenge. In this paper, we first revisit the key impact of task numbers on current MTRL method, and further reveal that naively expanding the parameters proves insufficient to counteract the performance degradation as the number of tasks escalates. Building upon these insights, we propose M3DT, a novel mixture-of-experts (MoE) framework that tackles task scalability by further unlocking the model's parameter scalability. Specifically, we enhance both the architecture and the optimization of the agent, where we strengthen the Decision Transformer (DT) backbone with MoE to reduce task load on parameter subsets, and introduce a three-stage training mechanism to facilitate efficient training with optimal performance. Experimental results show that, by increasing the number of experts, M3DT not only consistently enhances its performance as model expansion on the fixed task numbers, but also exhibits remarkable task scalability, successfully extending to 160 tasks with superior performance.