Perception-Control Coupled Visual Servoing for Textureless Objects Using Keypoint-Based EKF

TL;DR

This paper introduces a perception-control coupled visual servoing approach using keypoint-based EKF to improve accuracy and robustness in controlling textureless objects under challenging visual conditions.

Contribution

It proposes a novel integrated perception-control framework with probabilistic control law and EKF-based pose estimation for textureless object manipulation.

Findings

Outperforms traditional methods in accuracy

Enhances robustness under occlusions and poor visual conditions

Proven effective in real-world robotic grasping tasks

Abstract

Visual servoing is fundamental to robotic applications, enabling precise positioning and control. However, applying it to textureless objects remains a challenge due to the absence of reliable visual features. Moreover, adverse visual conditions, such as occlusions, often corrupt visual feedback, leading to reduced accuracy and instability in visual servoing. In this work, we build upon learning-based keypoint detection for textureless objects and propose a method that enhances robustness by tightly integrating perception and control in a closed loop. Specifically, we employ an Extended Kalman Filter (EKF) that integrates per-frame keypoint measurements to estimate 6D object pose, which drives pose-based visual servoing (PBVS) for control. The resulting camera motion, in turn, enhances the tracking of subsequent keypoints, effectively closing the perception-control loop. Additionally, unlike standard PBVS, we propose a probabilistic control law that computes both camera velocity and its associated uncertainty, enabling uncertainty-aware control for safe and reliable operation. We validate our approach on real-world robotic platforms using quantitative metrics and grasping experiments, demonstrating that our method outperforms traditional visual servoing techniques in both accuracy and practical application.