ReMoBot: Retrieval-Based Few-Shot Imitation Learning for Mobile Manipulation with Vision Foundation Models

TL;DR

ReMoBot introduces a retrieval-based few-shot imitation learning method utilizing vision foundation models to enable mobile robots to perform manipulation tasks effectively with limited demonstrations, demonstrating strong real-world performance and generalization.

Contribution

The paper presents ReMoBot, a novel retrieval-based few-shot imitation learning approach that leverages vision foundation models for mobile manipulation tasks, addressing data scarcity and partial observability issues.

Findings

ReMoBot achieves over 70% success rate on benchmark tasks with only 20 demonstrations.

ReMoBot outperforms baseline methods in real-world experiments without sim-to-real transfer.

The approach generalizes across different robot positions, object sizes, and materials.

Abstract

Imitation learning (IL) algorithms typically distill experience into parametric behavior policies to mimic expert demonstrations. However, with limited demonstrations, existing methods often struggle to generate accurate actions, particularly under partial observability. To address this problem, we introduce a few-shot IL approach, ReMoBot, which directly retrieves information from demonstrations to solve Mobile manipulation tasks with ego-centric visual observations. Given the current observation, ReMoBot utilizes vision foundation models to identify relevant demonstrations, considering visual similarity w.r.t. both individual observations and history trajectories. A motion selection policy then selects the proper command for the robot until the task is successfully completed. The performance of ReMoBot is evaluated on three mobile manipulation tasks with a Boston Dynamics Spot robot in both simulation and the real world. After benchmarking five approaches in simulation, we compare our method with two baselines in the real world, training directly on the real-world dataset without sim-to-real transfer. With only 20 demonstrations, ReMoBot outperforms the baselines, achieving high success rates in Table Uncover (70%) and Gap Cover (80%), while also showing promising performance on the more challenging Curtain Open task in the real-world setting. Furthermore, ReMoBot demonstrates generalization across varying robot positions, object sizes, and material types. Additional details are available at: https://sites.google.com/view/remobot/home