Aligning Motion-Blurred Images Using Contrastive Learning on Overcomplete Pixels

TL;DR

This paper introduces a contrastive learning approach to generate overcomplete pixel features that are invariant to motion blur, enabling effective alignment of video frames captured with moving cameras under challenging conditions.

Contribution

It presents a novel contrastive objective for overcomplete pixel features and demonstrates their effectiveness in aligning motion-blurred video frames.

Findings

U-Net trained with the proposed objective aligns frames in challenging videos

Overcomplete pixels encode object identity and pixel coordinates

Features are invariant to motion blur and other transformations

Abstract

We propose a new contrastive objective for learning overcomplete pixel-level features that are invariant to motion blur. Other invariances (e.g., pose, illumination, or weather) can be learned by applying the corresponding transformations on unlabeled images during self-supervised training. We showcase that a simple U-Net trained with our objective can produce local features useful for aligning the frames of an unseen video captured with a moving camera under realistic and challenging conditions. Using a carefully designed toy example, we also show that the overcomplete pixels can encode the identity of objects in an image and the pixel coordinates relative to these objects.