Hybrid Vision Servoing with Depp Alignment and GRU-Based Occlusion Recovery

TL;DR

This paper introduces a hybrid visual servoing system combining fast template matching, deep feature Lucas-Kanade, residual correction, and GRU-based occlusion prediction to achieve robust, real-time robot tracking under severe occlusions.

Contribution

It presents a novel hybrid framework that integrates multiple perception modules with a GRU-based predictor for robust, low-latency visual servoing during occlusions.

Findings

Maintains under 2px tracking error with up to 90% occlusion.

Operates at 30Hz for real-time control.

Demonstrates robustness in challenging scenarios.

Abstract

Vision-based control systems, such as image-based visual servoing (IBVS), have been extensively explored for precise robot manipulation. A persistent challenge, however, is maintaining robust target tracking under partial or full occlusions. Classical methods like Lucas-Kanade (LK) offer lightweight tracking but are fragile to occlusion and drift, while deep learning-based approaches often require continuous visibility and intensive computation. To address these gaps, we propose a hybrid visual tracking framework that bridges advanced perception with real-time servo control. First, a fast global template matcher constrains the pose search region; next, a deep-feature Lucas-Kanade module operating on early VGG layers refines alignment to sub-pixel accuracy (<2px); then, a lightweight residual regressor corrects local misalignments caused by texture degradation or partial occlusion. When visual confidence falls below a threshold, a GRU-based predictor seamlessly extrapolates pose updates from recent motion history. Crucially, the pipeline's final outputs-translation, rotation, and scale deltas-are packaged as direct control signals for 30Hz image-based servo loops. Evaluated on handheld video sequences with up to 90% occlusion, our system sustains under 2px tracking error, demonstrating the robustness and low-latency precision essential for reliable real-world robot vision applications.