Transformer Driven Visual Servoing for Fabric Texture Matching Using Dual-Arm Manipulator

TL;DR

This paper introduces a novel transformer-based visual servoing method for dual-arm robots to accurately align and place fabric textures, leveraging synthetic training for real-world application without prior texture data.

Contribution

It presents a new control scheme combining transformer-driven visual servoing with dual-arm impedance control and a novel attention module, enabling zero-shot fabric texture matching.

Findings

Accurate fabric alignment demonstrated in real-world tests.

Synthetic training enables zero-shot deployment.

Enhanced pose prediction accuracy with DEAM.

Abstract

In this paper, we propose a method to align and place a fabric piece on top of another using a dual-arm manipulator and a grayscale camera, so that their surface textures are accurately matched. We propose a novel control scheme that combines Transformer-driven visual servoing with dualarm impedance control. This approach enables the system to simultaneously control the pose of the fabric piece and place it onto the underlying one while applying tension to keep the fabric piece flat. Our transformer-based network incorporates pretrained backbones and a newly introduced Difference Extraction Attention Module (DEAM), which significantly enhances pose difference prediction accuracy. Trained entirely on synthetic images generated using rendering software, the network enables zero-shot deployment in real-world scenarios without requiring prior training on specific fabric textures. Real-world experiments demonstrate that the proposed system accurately aligns fabric pieces with different textures.