DIJE: Dense Image Jacobian Estimation for Robust Robotic Self-Recognition and Visual Servoing

TL;DR

This paper introduces DIJE, an efficient algorithm for pixel-wise image Jacobian estimation using optical flow and Kalman filtering, enabling robust self-recognition and visual control in robots without markers or prior structure knowledge.

Contribution

The paper presents a novel real-time, markerless image Jacobian estimation method that enhances robot self-recognition and visual servoing capabilities.

Findings

Successfully distinguished robot movement from external motion

Enabled real-time control of robot reaching and tool manipulation

Validated on a physical musculoskeletal robot

Abstract

For robots to move in the real world, they must first correctly understand the state of its own body and the tools that it holds. In this research, we propose DIJE, an algorithm to estimate the image Jacobian for every pixel. It is based on an optical flow calculation and a simplified Kalman Filter that can be efficiently run on the whole image in real time. It does not rely on markers nor knowledge of the robotic structure. We use the DIJE in a self-recognition process which can robustly distinguish between movement by the robot and by external entities, even when the motion overlaps. We also propose a visual servoing controller based on DIJE, which can learn to control the robot's body to conduct reaching movements or bimanual tool-tip control. The proposed algorithms were implemented on a physical musculoskeletal robot and its performance was verified. We believe that such global estimation of the visuomotor policy has the potential to be extended into a more general framework for manipulation.