A Two-stage Fine-tuning Strategy for Generalizable Manipulation Skill of Embodied AI

TL;DR

This paper introduces a two-stage fine-tuning strategy that significantly improves the generalization of Embodied AI models in manipulation tasks, demonstrated by winning the ManiSkill2 Challenge across all tracks.

Contribution

The paper presents a novel two-stage fine-tuning approach that enhances Embodied AI's ability to generalize to unseen scenarios, advancing the field beyond existing methods.

Findings

Achieved 1st place in all three tracks of the ManiSkill2 Challenge.

Demonstrated improved generalization in manipulation tasks.

Provided open-source code and models for reproducibility.

Abstract

The advent of Chat-GPT has led to a surge of interest in Embodied AI. However, many existing Embodied AI models heavily rely on massive interactions with training environments, which may not be practical in real-world situations. To this end, the Maniskill2 has introduced a full-physics simulation benchmark for manipulating various 3D objects. This benchmark enables agents to be trained using diverse datasets of demonstrations and evaluates their ability to generalize to unseen scenarios in testing environments. In this paper, we propose a novel two-stage fine-tuning strategy that aims to further enhance the generalization capability of our model based on the Maniskill2 benchmark. Through extensive experiments, we demonstrate the effectiveness of our approach by achieving the 1st prize in all three tracks of the ManiSkill2 Challenge. Our findings highlight the potential of our method to improve the generalization abilities of Embodied AI models and pave the way for their ractical applications in real-world scenarios. All codes and models of our solution is available at https://github.com/xtli12/GXU-LIPE.git