CLASP: General-Purpose Clothes Manipulation with Semantic Keypoints

TL;DR

CLASP introduces a semantic keypoint-based approach for versatile clothes manipulation, enabling robots to perform various tasks across diverse clothing types by combining vision language models and manipulation skills.

Contribution

It proposes a novel semantic keypoint representation for general-purpose clothes manipulation, bridging high-level planning and low-level execution across multiple tasks and clothing types.

Findings

Outperforms state-of-the-art methods in simulation across diverse clothes and tasks.

Demonstrates successful real-world application with a dual-arm robot.

Shows strong generalization capabilities in complex clothing manipulation tasks.

Abstract

Clothes manipulation, such as folding or hanging, is a critical capability for home service robots. Despite recent advances, most existing methods remain limited to specific clothes types and tasks, due to the complex, high-dimensional geometry of clothes. This paper presents CLothes mAnipulation with Semantic keyPoints (CLASP), which aims at general-purpose clothes manipulation over diverse clothes types, T-shirts, shorts, skirts, long dresses, ..., as well as different tasks, folding, flattening, hanging, .... The core idea of CLASP is semantic keypoints-e.g., ''left sleeve'' and ''right shoulder''-a sparse spatial-semantic representation, salient for both perception and action. Semantic keypoints of clothes can be reliably extracted from RGB-D images and provide an effective representation for a wide range of clothes manipulation policies. CLASP uses semantic keypoints as an intermediate representation to connect high-level task planning and low-level action execution. At the high level, it exploits vision language models (VLMs) to predict task plans over the semantic keypoints. At the low level, it executes the plans with the help of a set of pre-built manipulation skills conditioned on the keypoints. Extensive simulation experiments show that CLASP outperforms state-of-the-art baseline methods on multiple tasks across diverse clothes types, demonstrating strong performance and generalization. Further experiments with a Franka dual-arm system on four distinct tasks-folding, flattening, hanging, and placing-confirm CLASP's performance on real-life clothes manipulation.