Bridging Visual Affective Gap: Borrowing Textual Knowledge by Learning from Noisy Image-Text Pairs

TL;DR

This paper introduces a novel method called Partitioned Adaptive Contrastive Learning (PACL) that leverages textual knowledge from noisy social media data to bridge the affective gap in visual emotion recognition, significantly enhancing model performance.

Contribution

It proposes a new contrastive learning framework that separates sample types and exploits noisy image-text pairs to improve visual emotion recognition models.

Findings

Bridging the affective gap improves emotion recognition accuracy.

PACL outperforms existing methods on multiple downstream tasks.

Dynamic construction of positive and negative pairs enhances learning from noisy data.

Abstract

Visual emotion recognition (VER) is a longstanding field that has garnered increasing attention with the advancement of deep neural networks. Although recent studies have achieved notable improvements by leveraging the knowledge embedded within pre-trained visual models, the lack of direct association between factual-level features and emotional categories, called the "affective gap", limits the applicability of pre-training knowledge for VER tasks. On the contrary, the explicit emotional expression and high information density in textual modality eliminate the "affective gap". Therefore, we propose borrowing the knowledge from the pre-trained textual model to enhance the emotional perception of pre-trained visual models. We focus on the factual and emotional connections between images and texts in noisy social media data, and propose Partitioned Adaptive Contrastive Learning (PACL) to leverage these connections. Specifically, we manage to separate different types of samples and devise distinct contrastive learning strategies for each type. By dynamically constructing negative and positive pairs, we fully exploit the potential of noisy samples. Through comprehensive experiments, we demonstrate that bridging the "affective gap" significantly improves the performance of various pre-trained visual models in downstream emotion-related tasks. Our code is released on https://github.com/wdqqdw/PACL.