ClipCrop: Conditioned Cropping Driven by Vision-Language Model

TL;DR

This paper introduces ClipCrop, a novel image cropping method that leverages vision-language models to incorporate user intentions and improve generalization, especially in complex and ambiguous scenarios.

Contribution

It adapts a transformer decoder with a pre-trained CLIP-based detection model to enable text or image-guided cropping, enhancing robustness and user control.

Findings

Effective user-intentional cropping guided by text/image queries.

Improved generalization on ambiguous and complex images.

Successful learning of aesthetic cropping with limited data.

Abstract

Image cropping has progressed tremendously under the data-driven paradigm. However, current approaches do not account for the intentions of the user, which is an issue especially when the composition of the input image is complex. Moreover, labeling of cropping data is costly and hence the amount of data is limited, leading to poor generalization performance of current algorithms in the wild. In this work, we take advantage of vision-language models as a foundation for creating robust and user-intentional cropping algorithms. By adapting a transformer decoder with a pre-trained CLIP-based detection model, OWL-ViT, we develop a method to perform cropping with a text or image query that reflects the user's intention as guidance. In addition, our pipeline design allows the model to learn text-conditioned aesthetic cropping with a small cropping dataset, while inheriting the open-vocabulary ability acquired from millions of text-image pairs. We validate our model through extensive experiments on existing datasets as well as a new cropping test set we compiled that is characterized by content ambiguity.