GeCo-SRT: Geometry-aware Continual Adaptation for Robotic Cross-Task Sim-to-Real Transfer

TL;DR

GeCo-SRT introduces a geometry-aware continual adaptation framework that leverages local geometric features to improve and accelerate robotic sim-to-real transfer across multiple tasks, reducing data needs and enhancing robustness.

Contribution

The paper presents a novel continual transfer paradigm with a geometry-aware mixture-of-experts and prioritized experience replay, enabling knowledge accumulation and improved adaptation efficiency.

Findings

Achieves 52% average performance improvement over baseline.

Demonstrates significant data efficiency, requiring only 1/6 of data for new tasks.

Effectively maintains cross-task performance through expert specialization.

Abstract

Bridging the sim-to-real gap is important for applying low-cost simulation data to real-world robotic systems. However, previous methods are severely limited by treating each transfer as an isolated endeavor, demanding repeated, costly tuning and wasting prior transfer experience. To move beyond isolated sim-to-real, we build a continual cross-task sim-to-real transfer paradigm centered on knowledge accumulation across iterative transfers, thereby enabling effective and efficient adaptation to novel tasks. Thus, we propose GeCo-SRT, a geometry-aware continual adaptation method. It utilizes domain-invariant and task-invariant knowledge from local geometric features as a transferable foundation to accelerate adaptation during subsequent sim-to-real transfers. This method starts with a geometry-aware mixture-of-experts module, which dynamically activates experts to specialize in distinct geometric knowledge to bridge observation sim-to-real gap. Further, the geometry-expert-guided prioritized experience replay module preferentially samples from underutilized experts, refreshing specialized knowledge to combat forgetting and maintain robust cross-task performance. Leveraging knowledge accumulated during iterative transfer, GeCo-SRT method not only achieves 52% average performance improvement over the baseline, but also demonstrates significant data efficiency for new task adaptation with only 1/6 data. We hope this work inspires approaches for efficient, low-cost cross-task sim-to-real transfer.