CDG-MAE: Learning Correspondences from Diffusion Generated Views

TL;DR

CDG-MAE introduces a self-supervised learning approach that generates diverse synthetic views from static images using diffusion models, enabling improved dense correspondence learning without extensive video data.

Contribution

The paper proposes a novel MAE-based method utilizing synthetic views from diffusion models, enhancing self-supervised correspondence learning beyond traditional image crops and reducing reliance on video datasets.

Findings

Outperforms existing image-based MAE methods in correspondence tasks.

Effectively narrows the performance gap between image-based and video-based approaches.

Demonstrates the effectiveness of synthetic view generation for self-supervised learning.

Abstract

Learning dense correspondences, critical for application such as video label propagation, is hindered by tedious and unscalable manual annotation. Self-supervised methods address this by using a cross-view pretext task, often modeled with a masked autoencoder, where a masked target view is reconstructed from an anchor view. However, acquiring effective training data remains a challenge - collecting diverse video datasets is difficult and costly, while simple image crops lack necessary pose variations. This paper introduces CDG-MAE, a novel MAE-based self-supervised method that uses diverse synthetic views generated from static images via an image-conditioned diffusion model. These generated views exhibit substantial changes in pose and perspective, providing a rich training signal that overcomes the limitations of video and crop-based anchors. We present a quantitative method to evaluate local and global consistency of generated images, discussing their use for cross-view self-supervised pretraining. Furthermore, we enhance the standard single-anchor MAE setting to a multi-anchor strategy to effectively modulate the difficulty of pretext task. CDG-MAE significantly outperforms state-of-the-art MAE methods reliant only on images and substantially narrows the performance gap to video-based approaches.