MixDiff: Mixing Natural and Synthetic Images for Robust Self-Supervised Representations

TL;DR

MixDiff is a self-supervised learning framework that combines real and synthetic images to improve robustness and domain transfer capabilities of learned representations, achieving significant accuracy gains.

Contribution

It introduces a novel approach to mix real and synthetic images in SSL, enhancing robustness without relying on traditional augmentations.

Findings

Boosts SimCLR accuracy by 4.56% on ImageNet-1K

Improves robustness across various datasets

Achieves competitive performance without augmentations

Abstract

This paper introduces MixDiff, a new self-supervised learning (SSL) pre-training framework that combines real and synthetic images. Unlike traditional SSL methods that predominantly use real images, MixDiff uses a variant of Stable Diffusion to replace an augmented instance of a real image, facilitating the learning of cross real-synthetic image representations. Our key insight is that while models trained solely on synthetic images underperform, combining real and synthetic data leads to more robust and adaptable representations. Experiments show MixDiff enhances SimCLR, BarlowTwins, and DINO across various robustness datasets and domain transfer tasks, boosting SimCLR's ImageNet-1K accuracy by 4.56%. Our framework also demonstrates comparable performance without needing any augmentations, a surprising finding in SSL where augmentations are typically crucial.