Active Annotation of Informative Overlapping Frames in Video Mosaicking Applications

TL;DR

This paper presents an efficient active annotation framework for long-range frame registration in video mosaicking, reducing computational costs and improving global scene consistency by selectively querying informative frame pairs.

Contribution

It introduces a novel iterative strategy for selecting informative frame pairs for annotation, combining reward-based selection with online adaptable overlap models, applicable to various mosaicking scenarios.

Findings

Improves registration accuracy in long sequences.

Reduces computational costs by selective annotation.

Provides ground truth landmarks for evaluation.

Abstract

Video mosaicking requires the registration of overlapping frames located at distant timepoints in the sequence to ensure global consistency of the reconstructed scene. However, fully automated registration of such long-range pairs is (i) challenging when the registration of images itself is difficult; and (ii) computationally expensive for long sequences due to the large number of candidate pairs for registration. In this paper, we introduce an efficient framework for the active annotation of long-range pairwise correspondences in a sequence. Our framework suggests pairs of images that are sought to be informative to an oracle agent (e.g., a human user, or a reliable matching algorithm) who provides visual correspondences on each suggested pair. Informative pairs are retrieved according to an iterative strategy based on a principled annotation reward coupled with two complementary and online adaptable models of frame overlap. In addition to the efficient construction of a mosaic, our framework provides, as a by-product, ground truth landmark correspondences which can be used for evaluation or learning purposes. We evaluate our approach in both automated and interactive scenarios via experiments on synthetic sequences, on a publicly available dataset for aerial imaging and on a clinical dataset for placenta mosaicking during fetal surgery.