TieBot: Learning to Knot a Tie from Visual Demonstration through a Real-to-Sim-to-Real Approach

TL;DR

TieBot is a robotic system that learns to knot a tie from visual demonstrations by combining simulation and real-world training, effectively handling high deformation and long-horizon manipulation tasks.

Contribution

The paper introduces a novel Hierarchical Feature Matching approach and a Real-to-Sim-to-Real learning pipeline for tie-knotting from visual demonstrations.

Findings

Successfully knots a tie in real-world experiments with a 50% success rate.

Demonstrates effective transfer from simulation to real-world execution.

Validates the approach in both simulation and physical robot settings.

Abstract

The tie-knotting task is highly challenging due to the tie's high deformation and long-horizon manipulation actions. This work presents TieBot, a Real-to-Sim-to-Real learning from visual demonstration system for the robots to learn to knot a tie. We introduce the Hierarchical Feature Matching approach to estimate a sequence of tie's meshes from the demonstration video. With these estimated meshes used as subgoals, we first learn a teacher policy using privileged information. Then, we learn a student policy with point cloud observation by imitating teacher policy. Lastly, our pipeline applies learned policy to real-world execution. We demonstrate the effectiveness of TieBot in simulation and the real world. In the real-world experiment, a dual-arm robot successfully knots a tie, achieving 50% success rate among 10 trials. Videos can be found https://tiebots.github.io/.