High-Level Concepts for Affective Understanding of Images

TL;DR

This paper introduces a novel framework using high-level concepts derived from pretrained CNNs to better understand and predict emotional responses to images, bridging the gap between image content and viewer emotion.

Contribution

It presents a linear and nonlinear modeling approach that explicitly links high-level concepts to emotions, improving interpretability and prediction accuracy over previous methods.

Findings

The model effectively associates HLCs with specific emotional classes.

The nonlinear SVR model improves emotion prediction accuracy.

Results are comparable to existing state-of-the-art methods.

Abstract

This paper aims to bridge the affective gap between image content and the emotional response of the viewer it elicits by using High-Level Concepts (HLCs). In contrast to previous work that relied solely on low-level features or used convolutional neural network (CNN) as a black-box, we use HLCs generated by pretrained CNNs in an explicit way to investigate the relations/associations between these HLCs and a (small) set of Ekman's emotional classes. As a proof-of-concept, we first propose a linear admixture model for modeling these relations, and the resulting computational framework allows us to determine the associations between each emotion class and certain HLCs (objects and places). This linear model is further extended to a nonlinear model using support vector regression (SVR) that aims to predict the viewer's emotional response using both low-level image features and HLCs extracted from images. These class-specific regressors are then assembled into a regressor ensemble that provide a flexible and effective predictor for predicting viewer's emotional responses from images. Experimental results have demonstrated that our results are comparable to existing methods, with a clear view of the association between HLCs and emotional classes that is ostensibly missing in most existing work.