PDANet: Polarity-consistent Deep Attention Network for Fine-grained Visual Emotion Regression

TL;DR

This paper introduces PDANet, a novel deep learning architecture that incorporates attention mechanisms and emotion polarity constraints to improve fine-grained visual emotion regression, outperforming existing methods.

Contribution

The paper proposes a new CNN-based network with integrated spatial and channel-wise attention and a polarity-consistent regression loss for enhanced emotion analysis.

Findings

PDANet outperforms state-of-the-art methods on multiple datasets.

Incorporating polarity constraints improves attention quality and regression accuracy.

The model effectively captures subtle emotional nuances in images.

Abstract

Existing methods on visual emotion analysis mainly focus on coarse-grained emotion classification, i.e. assigning an image with a dominant discrete emotion category. However, these methods cannot well reflect the complexity and subtlety of emotions. In this paper, we study the fine-grained regression problem of visual emotions based on convolutional neural networks (CNNs). Specifically, we develop a Polarity-consistent Deep Attention Network (PDANet), a novel network architecture that integrates attention into a CNN with an emotion polarity constraint. First, we propose to incorporate both spatial and channel-wise attentions into a CNN for visual emotion regression, which jointly considers the local spatial connectivity patterns along each channel and the interdependency between different channels. Second, we design a novel regression loss, i.e. polarity-consistent regression (PCR) loss, based on the weakly supervised emotion polarity to guide the attention generation. By optimizing the PCR loss, PDANet can generate a polarity preserved attention map and thus improve the emotion regression performance. Extensive experiments are conducted on the IAPS, NAPS, and EMOTIC datasets, and the results demonstrate that the proposed PDANet outperforms the state-of-the-art approaches by a large margin for fine-grained visual emotion regression. Our source code is released at: https://github.com/ZizhouJia/PDANet.