Whole-Body Mobile Manipulation using Offline Reinforcement Learning on Sub-optimal Controllers

TL;DR

This paper introduces WHOLE-MoMa, a two-stage offline reinforcement learning pipeline that improves whole-body mobile manipulation by leveraging sub-optimal controllers for data collection and policy refinement, enabling effective real-world task execution.

Contribution

It proposes a novel offline RL approach that uses sub-optimal controllers for data generation and extends implicit Q-learning with Q-chunking for complex coordination tasks.

Findings

Outperforms baseline methods in simulation tasks.

Achieves high success rates in real robot experiments.

Policies transfer directly without finetuning.

Abstract

Mobile Manipulation (MoMa) of articulated objects, such as opening doors, drawers, and cupboards, demands simultaneous, whole-body coordination between a robot's base and arms. Classical whole-body controllers (WBCs) can solve such problems via hierarchical optimization, but require extensive hand-tuned optimization and remain brittle. Learning-based methods, on the other hand, show strong generalization capabilities but typically rely on expensive whole-body teleoperation data or heavy reward engineering. We observe that even a sub-optimal WBC is a powerful structural prior: it can be used to collect data in a constrained, task-relevant region of the state-action space, and its behavior can still be improved upon using offline reinforcement learning. Building on this, we propose WHOLE-MoMa, a two-stage pipeline that first generates diverse demonstrations by randomizing a lightweight WBC, and then applies offline RL to identify and stitch together improved behaviors via a reward signal. To support the expressive action-chunked diffusion policies needed for complex coordination tasks, we extend offline implicit Q-learning with Q-chunking for chunk-level critic evaluation and advantage-weighted policy extraction. On three tasks of increasing difficulty using a TIAGo++ mobile manipulator in simulation, WHOLE-MoMa significantly outperforms WBC, behavior cloning, and several offline RL baselines. Policies transfer directly to the real robot without finetuning, achieving 80% success in bimanual drawer manipulation and 68% in simultaneous cupboard opening and object placement, all without any teleoperated or real-world training data.