KOVIS: Keypoint-based Visual Servoing with Zero-Shot Sim-to-Real Transfer for Robotics Manipulation

TL;DR

KOVIS is a novel visual servoing approach that trains entirely in simulation and transfers directly to real robots for precise manipulation tasks using keypoints and deep learning.

Contribution

It introduces a calibration-free, deep learning-based visual servoing method trained solely in simulation for zero-shot sim-to-real transfer in robotic manipulation.

Findings

Successful zero-shot transfer from simulation to real robots.

Effective in tasks like grasping, peg-in-hole, and screw insertion.

Achieves high precision with 4mm clearance in insertion tasks.

Abstract

We present KOVIS, a novel learning-based, calibration-free visual servoing method for fine robotic manipulation tasks with eye-in-hand stereo camera system. We train the deep neural network only in the simulated environment; and the trained model could be directly used for real-world visual servoing tasks. KOVIS consists of two networks. The first keypoint network learns the keypoint representation from the image using with an autoencoder. Then the visual servoing network learns the motion based on keypoints extracted from the camera image. The two networks are trained end-to-end in the simulated environment by self-supervised learning without manual data labeling. After training with data augmentation, domain randomization, and adversarial examples, we are able to achieve zero-shot sim-to-real transfer to real-world robotic manipulation tasks. We demonstrate the effectiveness of the proposed method in both simulated environment and real-world experiment with different robotic manipulation tasks, including grasping, peg-in-hole insertion with 4mm clearance, and M13 screw insertion. The demo video is available at http://youtu.be/gfBJBR2tDzA