Learning Image Matching by Simply Watching Video

TL;DR

This paper introduces an unsupervised method for image matching by training a CNN for frame-interpolation on videos and then deriving correspondences through inversion, achieving performance comparable to traditional methods.

Contribution

It presents a novel unsupervised approach to image matching that leverages video data and analysis-by-synthesis, avoiding manual annotations.

Findings

Achieves competitive accuracy with traditional methods

Uses only video data for training, no annotations needed

Demonstrates effectiveness on standard benchmarks

Abstract

This work presents an unsupervised learning based approach to the ubiquitous computer vision problem of image matching. We start from the insight that the problem of frame-interpolation implicitly solves for inter-frame correspondences. This permits the application of analysis-by-synthesis: we firstly train and apply a Convolutional Neural Network for frame-interpolation, then obtain correspondences by inverting the learned CNN. The key benefit behind this strategy is that the CNN for frame-interpolation can be trained in an unsupervised manner by exploiting the temporal coherency that is naturally contained in real-world video sequences. The present model therefore learns image matching by simply watching videos. Besides a promise to be more generally applicable, the presented approach achieves surprising performance comparable to traditional empirically designed methods.