Learning Subject-Aware Cropping by Outpainting Professional Photos

TL;DR

This paper introduces GenCrop, a weakly-supervised method that learns subject-aware image cropping from professional photos by combining stock images and a diffusion model to generate training data without manual annotations.

Contribution

GenCrop is the first weakly-supervised approach to subject-aware cropping that leverages existing stock images and a diffusion model to generate training pairs without manual labels.

Findings

GenCrop performs competitively with state-of-the-art supervised methods.

It significantly outperforms other weakly-supervised baselines.

The method effectively learns subject-aware cropping without manual annotations.

Abstract

How to frame (or crop) a photo often depends on the image subject and its context; e.g., a human portrait. Recent works have defined the subject-aware image cropping task as a nuanced and practical version of image cropping. We propose a weakly-supervised approach (GenCrop) to learn what makes a high-quality, subject-aware crop from professional stock images. Unlike supervised prior work, GenCrop requires no new manual annotations beyond the existing stock image collection. The key challenge in learning from this data, however, is that the images are already cropped and we do not know what regions were removed. Our insight is to combine a library of stock images with a modern, pre-trained text-to-image diffusion model. The stock image collection provides diversity and its images serve as pseudo-labels for a good crop, while the text-image diffusion model is used to out-paint (i.e., outward inpainting) realistic uncropped images. Using this procedure, we are able to automatically generate a large dataset of cropped-uncropped training pairs to train a cropping model. Despite being weakly-supervised, GenCrop is competitive with state-of-the-art supervised methods and significantly better than comparable weakly-supervised baselines on quantitative and qualitative evaluation metrics.