Free-ATM: Exploring Unsupervised Learning on Diffusion-Generated Images with Free Attention Masks

TL;DR

This paper introduces a novel unsupervised learning approach leveraging free attention masks derived from diffusion models' cross-attention layers, significantly improving performance on various vision tasks using synthetic images.

Contribution

It uncovers the inherent attention masks in diffusion models and exploits them to enhance multiple unsupervised learning techniques on generated images.

Findings

Improved image classification accuracy

Enhanced detection and segmentation performance

Close gap between synthetic and real data pretraining

Abstract

Despite the rapid advancement of unsupervised learning in visual representation, it requires training on large-scale datasets that demand costly data collection, and pose additional challenges due to concerns regarding data privacy. Recently, synthetic images generated by text-to-image diffusion models, have shown great potential for benefiting image recognition. Although promising, there has been inadequate exploration dedicated to unsupervised learning on diffusion-generated images. To address this, we start by uncovering that diffusion models' cross-attention layers inherently provide annotation-free attention masks aligned with corresponding text inputs on generated images. We then investigate the problems of three prevalent unsupervised learning techniques ( i.e., contrastive learning, masked modeling, and vision-language pretraining) and introduce customized solutions by fully exploiting the aforementioned free attention masks. Our approach is validated through extensive experiments that show consistent improvements in baseline models across various downstream tasks, including image classification, detection, segmentation, and image-text retrieval. By utilizing our method, it is possible to close the performance gap between unsupervised pretraining on synthetic data and real-world scenarios.