Learning the Generalizable Manipulation Skills on Soft-body Tasks via Guided Self-attention Behavior Cloning Policy

TL;DR

This paper introduces GP2E, a behavior cloning policy that enhances generalizable manipulation skills for soft-body tasks by integrating semantic features, guided self-attention, and a two-stage fine-tuning strategy, demonstrated by winning a challenge.

Contribution

The work presents a novel policy architecture combining semantic feature extraction, guided self-attention, and two-stage fine-tuning for soft-body manipulation tasks, advancing generalization in Embodied AI.

Findings

Achieved 1st place in the ManiSkill2 Challenge soft-body track.

Demonstrated improved generalization in soft-body manipulation tasks.

Validated effectiveness through extensive experiments.

Abstract

Embodied AI represents a paradigm in AI research where artificial agents are situated within and interact with physical or virtual environments. Despite the recent progress in Embodied AI, it is still very challenging to learn the generalizable manipulation skills that can handle large deformation and topological changes on soft-body objects, such as clay, water, and soil. In this work, we proposed an effective policy, namely GP2E behavior cloning policy, which can guide the agent to learn the generalizable manipulation skills from soft-body tasks, including pouring, filling, hanging, excavating, pinching, and writing. Concretely, we build our policy from three insights:(1) Extracting intricate semantic features from point cloud data and seamlessly integrating them into the robot's end-effector frame; (2) Capturing long-distance interactions in long-horizon tasks through the incorporation of our guided self-attention module; (3) Mitigating overfitting concerns and facilitating model convergence to higher accuracy levels via the introduction of our two-stage fine-tuning strategy. Through extensive experiments, we demonstrate the effectiveness of our approach by achieving the 1st prize in the soft-body track of the ManiSkill2 Challenge at the CVPR 2023 4th Embodied AI workshop. Our findings highlight the potential of our method to improve the generalization abilities of Embodied AI models and pave the way for their practical applications in real-world scenarios.