Deep Aesthetic Quality Assessment with Semantic Information

TL;DR

This paper proposes a multi-task deep learning framework that integrates semantic recognition to improve automatic aesthetic quality assessment of images, achieving state-of-the-art results by leveraging inter-task correlations.

Contribution

It introduces a novel multi-task CNN model with inter-task relationship learning that effectively combines aesthetic and semantic recognition tasks.

Findings

Semantic recognition enhances aesthetic assessment accuracy.

The multi-task model outperforms previous methods on AVA and Photo.net datasets.

Inter-task correlation learning improves aesthetic feature representation.

Abstract

Human beings often assess the aesthetic quality of an image coupled with the identification of the image's semantic content. This paper addresses the correlation issue between automatic aesthetic quality assessment and semantic recognition. We cast the assessment problem as the main task among a multi-task deep model, and argue that semantic recognition task offers the key to address this problem. Based on convolutional neural networks, we employ a single and simple multi-task framework to efficiently utilize the supervision of aesthetic and semantic labels. A correlation item between these two tasks is further introduced to the framework by incorporating the inter-task relationship learning. This item not only provides some useful insight about the correlation but also improves assessment accuracy of the aesthetic task. Particularly, an effective strategy is developed to keep a balance between the two tasks, which facilitates to optimize the parameters of the framework. Extensive experiments on the challenging AVA dataset and Photo.net dataset validate the importance of semantic recognition in aesthetic quality assessment, and demonstrate that multi-task deep models can discover an effective aesthetic representation to achieve state-of-the-art results.