Rapidly Adapting Policies to the Real World via Simulation-Guided Fine-Tuning

TL;DR

This paper presents a simulation-guided fine-tuning framework that leverages simulation-derived value functions to efficiently adapt robot policies in the real world, significantly reducing data requirements and improving success in complex manipulation tasks.

Contribution

The paper introduces SGFT, a novel method that uses simulation-based value functions to guide real-world exploration, enhancing sample efficiency and success in challenging tasks.

Findings

SGFT outperforms baseline fine-tuning methods by up to ten times in real-world sample efficiency.

SGFT succeeds in tasks where zero-shot sim-to-real transfer fails.

Theoretical analysis supports the effectiveness of simulation-guided exploration.

Abstract

Robot learning requires a considerable amount of high-quality data to realize the promise of generalization. However, large data sets are costly to collect in the real world. Physics simulators can cheaply generate vast data sets with broad coverage over states, actions, and environments. However, physics engines are fundamentally misspecified approximations to reality. This makes direct zero-shot transfer from simulation to reality challenging, especially in tasks where precise and force-sensitive manipulation is necessary. Thus, fine-tuning these policies with small real-world data sets is an appealing pathway for scaling robot learning. However, current reinforcement learning fine-tuning frameworks leverage general, unstructured exploration strategies which are too inefficient to make real-world adaptation practical. This paper introduces the Simulation-Guided Fine-tuning (SGFT) framework, which demonstrates how to extract structural priors from physics simulators to substantially accelerate real-world adaptation. Specifically, our approach uses a value function learned in simulation to guide real-world exploration. We demonstrate this approach across five real-world dexterous manipulation tasks where zero-shot sim-to-real transfer fails. We further demonstrate our framework substantially outperforms baseline fine-tuning methods, requiring up to an order of magnitude fewer real-world samples and succeeding at difficult tasks where prior approaches fail entirely. Last but not least, we provide theoretical justification for this new paradigm which underpins how SGFT can rapidly learn high-performance policies in the face of large sim-to-real dynamics gaps. Project webpage: https://weirdlabuw.github.io/sgft/{weirdlabuw.github.io/sgft}