Sim-to-Real Dynamic Object Manipulation on Conveyor Systems via Optimization Path Shaping

TL;DR

This paper presents GEM, a geometry-focused model that improves sim-to-real transfer for dynamic object manipulation on conveyor systems, achieving high success rates in real-world deployment without scene-specific data.

Contribution

The introduction of GEM with appearance noise annealing enhances imitation learning from simulated data, enabling robust generalization in real-world conveyor tasks.

Findings

GEM achieves over 97% success rate in real-world tableware collection.

It generalizes across different backgrounds, robot types, and object geometries.

Extensive experiments validate its effectiveness in simulated and real environments.

Abstract

Realizing generalizable dynamic object manipulation on conveyor systems is important for enhancing manufacturing efficiency, as it eliminates specialized engineering for different scenarios. To this end, imitation learning emerges as a promising paradigm, leveraging expert demonstrations to teach a policy manipulation skills. Although the generalization of an imitation learning policy can be improved by increasing demonstrations, demonstration collection is labor-intensive. Besides, public dynamic object manipulation data is scarce. In this work, we address this data scarcity problem via generating demonstrations in a simulator. A significant challenge of using simulated data lies in the appearance gap between simulated and real-world observations. To tackle this challenge, we propose Geometry-Enhanced Model (GEM), which employs our designed appearance noise annealing strategy to shape the policy optimization path, thereby prioritizing the geometry information in observations. Extensive experiments in simulated and real-world tasks demonstrate that GEM can generalize across environment backgrounds, robot embodiments, motion dynamics, and object geometries. Notably, GEM is deployed in a real canteen for tableware collection. Without test-scene data, GEM achieves a success rate of over 97% across more than 10,000 operations.