Semantic Image Cropping

TL;DR

This paper introduces semantic image cropping, a novel approach that enhances image relevance for specific entities by incorporating semantic information, supported by a new dataset and a deep learning system.

Contribution

It presents the concept of semantic image cropping, a new dataset with ground truth annotations, and a deep learning model that considers semantics for improved cropping.

Findings

State-of-the-art aesthetic cropping algorithms perform poorly on semantic cropping.

The proposed deep learning system outperforms existing methods in semantic relevance.

Semantic information improves the quality of image croppings for specific entities.

Abstract

Automatic image cropping techniques are commonly used to enhance the aesthetic quality of an image; they do it by detecting the most beautiful or the most salient parts of the image and removing the unwanted content to have a smaller image that is more visually pleasing. In this thesis, I introduce an additional dimension to the problem of cropping, semantics. I argue that image cropping can also enhance the image's relevancy for a given entity by using the semantic information contained in the image. I call this problem, Semantic Image Cropping. To support my argument, I provide a new dataset containing 100 images with at least two different entities per image and four ground truth croppings collected using Amazon Mechanical Turk. I use this dataset to show that state-of-the-art cropping algorithms that only take into account aesthetics do not perform well in the problem of semantic image cropping. Additionally, I provide a new deep learning system that takes not just aesthetics but also semantics into account to generate image croppings, and I evaluate its performance using my new semantic cropping dataset, showing that using the semantic information of an image can help to produce better croppings.