ImageSubject: A Large-scale Dataset for Subject Detection

TL;DR

This paper introduces ImageSubject, a large-scale dataset of 107,700 images from movie shots for training models to detect main subjects, highlighting the importance of understanding image layout and context.

Contribution

The creation of ImageSubject, the first dataset focused on localizing the main subject in images, with detailed analysis and comparison to related tasks like saliency and object detection.

Findings

Transformer-based models perform best on subject detection.

The dataset enables better understanding of image layout and context.

Subject detection differs from traditional object detection and saliency tasks.

Abstract

Main subjects usually exist in the images or videos, as they are the objects that the photographer wants to highlight. Human viewers can easily identify them but algorithms often confuse them with other objects. Detecting the main subjects is an important technique to help machines understand the content of images and videos. We present a new dataset with the goal of training models to understand the layout of the objects and the context of the image then to find the main subjects among them. This is achieved in three aspects. By gathering images from movie shots created by directors with professional shooting skills, we collect the dataset with strong diversity, specifically, it contains 107\,700 images from 21\,540 movie shots. We labeled them with the bounding box labels for two classes: subject and non-subject foreground object. We present a detailed analysis of the dataset and compare the task with saliency detection and object detection. ImageSubject is the first dataset that tries to localize the subject in an image that the photographer wants to highlight. Moreover, we find the transformer-based detection model offers the best result among other popular model architectures. Finally, we discuss the potential applications and conclude with the importance of the dataset.