Abstracting Robot Manipulation Skills via Mixture-of-Experts Diffusion Policies

TL;DR

This paper introduces SMP, a diffusion-based mixture-of-experts policy for robot manipulation that efficiently learns and composes skills, enabling scalable multi-task learning with high success and low inference cost.

Contribution

The paper proposes a novel SMP framework that uses a mixture-of-experts approach with sticky routing and variational training for scalable multi-task robot manipulation.

Findings

SMP achieves higher success rates than large diffusion baselines.

SMP has significantly lower inference costs.

SMP demonstrates effective transfer learning in real robot experiments.

Abstract

Diffusion-based policies have recently shown strong results in robot manipulation, but their extension to multi-task scenarios is hindered by the high cost of scaling model size and demonstrations. We introduce Skill Mixture-of-Experts Policy (SMP), a diffusion-based mixture-of-experts policy that learns a compact orthogonal skill basis and uses sticky routing to compose actions from a small, task-relevant subset of experts at each step. A variational training objective supports this design, and adaptive expert activation at inference yields fast sampling without oversized backbones. We validate SMP in simulation and on a real dual-arm platform with multi-task learning and transfer learning tasks, where SMP achieves higher success rates and markedly lower inference cost than large diffusion baselines. These results indicate a practical path toward scalable, transferable multi-task manipulation: learn reusable skills once, activate only what is needed, and adapt quickly when tasks change.