Neural Congealing: Aligning Images to a Joint Semantic Atlas

TL;DR

Neural Congealing is a zero-shot, self-supervised framework that aligns images to a shared semantic atlas using pre-trained features, handling diverse variations without additional annotations.

Contribution

It introduces a novel self-supervised method leveraging DINO-ViT features to jointly align images to a semantic atlas without requiring training data or masks.

Findings

Effective alignment across diverse image sets

Outperforms state-of-the-art methods without extensive training

Handles severe variations in appearance, pose, and background

Abstract

We present Neural Congealing -- a zero-shot self-supervised framework for detecting and jointly aligning semantically-common content across a given set of images. Our approach harnesses the power of pre-trained DINO-ViT features to learn: (i) a joint semantic atlas -- a 2D grid that captures the mode of DINO-ViT features in the input set, and (ii) dense mappings from the unified atlas to each of the input images. We derive a new robust self-supervised framework that optimizes the atlas representation and mappings per image set, requiring only a few real-world images as input without any additional input information (e.g., segmentation masks). Notably, we design our losses and training paradigm to account only for the shared content under severe variations in appearance, pose, background clutter or other distracting objects. We demonstrate results on a plethora of challenging image sets including sets of mixed domains (e.g., aligning images depicting sculpture and artwork of cats), sets depicting related yet different object categories (e.g., dogs and tigers), or domains for which large-scale training data is scarce (e.g., coffee mugs). We thoroughly evaluate our method and show that our test-time optimization approach performs favorably compared to a state-of-the-art method that requires extensive training on large-scale datasets.