DexKnot: Generalizable Visuomotor Policy Learning for Dexterous Bag-Knotting Manipulation

TL;DR

DexKnot introduces a novel framework combining keypoint affordance and diffusion policy to enable robots to generalize bag-knotting tasks across diverse bag shapes and deformations, overcoming previous limitations.

Contribution

The paper presents DexKnot, a shape-agnostic, keypoint-based approach that improves robot generalization in complex bag-knotting tasks using diffusion transformers and real-world data.

Findings

Achieves reliable knotting on unseen bag instances

Reduces observation complexity via keypoint representation

Demonstrates effective generalization across diverse deformations

Abstract

Knotting plastic bags is a common task in daily life, yet it is challenging for robots due to the bags' infinite degrees of freedom and complex physical dynamics. Existing methods often struggle in generalization to unseen bag instances or deformations. To address this, we present DexKnot, a framework that combines keypoint affordance with diffusion policy to learn a generalizable bag-knotting policy. Our approach learns a shape-agnostic representation of bags from keypoint correspondence data collected through real-world manual deformation. For an unseen bag configuration, the keypoints can be identified by matching the representation to a reference. These keypoints are then provided to a diffusion transformer, which generates robot action based on a small number of human demonstrations. DexKnot enables effective policy generalization by reducing the dimensionality of observation space into a sparse set of keypoints. Experiments show that DexKnot achieves reliable and consistent knotting performance across a variety of previously unseen instances and deformations.