HVC-Net: Unifying Homography, Visibility, and Confidence Learning for Planar Object Tracking

TL;DR

HVC-Net introduces a unified CNN framework that jointly learns homography, visibility, and confidence to enhance the robustness and accuracy of planar object tracking in videos, outperforming existing methods.

Contribution

The paper presents a novel CNN model integrating homography, visibility, and confidence modules within a Lucas-Kanade tracking pipeline for improved planar object tracking.

Findings

Outperforms state-of-the-art on POT and TMT datasets.

Effectively handles appearance changes, occlusions, and motions.

Demonstrates success in real-world in-video advertisement synthesis.

Abstract

Robust and accurate planar tracking over a whole video sequence is vitally important for many vision applications. The key to planar object tracking is to find object correspondences, modeled by homography, between the reference image and the tracked image. Existing methods tend to obtain wrong correspondences with changing appearance variations, camera-object relative motions and occlusions. To alleviate this problem, we present a unified convolutional neural network (CNN) model that jointly considers homography, visibility, and confidence. First, we introduce correlation blocks that explicitly account for the local appearance changes and camera-object relative motions as the base of our model. Second, we jointly learn the homography and visibility that links camera-object relative motions with occlusions. Third, we propose a confidence module that actively monitors the estimation quality from the pixel correlation distributions obtained in correlation blocks. All these modules are plugged into a Lucas-Kanade (LK) tracking pipeline to obtain both accurate and robust planar object tracking. Our approach outperforms the state-of-the-art methods on public POT and TMT datasets. Its superior performance is also verified on a real-world application, synthesizing high-quality in-video advertisements.