SentiFormer: Metadata Enhanced Transformer for Image Sentiment Analysis

TL;DR

SentiFormer introduces a novel transformer-based framework that effectively integrates multiple metadata types with images for improved sentiment analysis, outperforming existing methods on public datasets.

Contribution

The paper proposes a new metadata-enhanced transformer model that adaptively learns and fuses diverse metadata with images for sentiment analysis.

Findings

Outperforms existing methods on three datasets

Effectively integrates multiple metadata types

Demonstrates superior accuracy in sentiment prediction

Abstract

As more and more internet users post images online to express their daily emotions, image sentiment analysis has attracted increasing attention. Recently, researchers generally tend to design different neural networks to extract visual features from images for sentiment analysis. Despite the significant progress, metadata, the data (e.g., text descriptions and keyword tags) for describing the image, has not been sufficiently explored in this task. In this paper, we propose a novel Metadata Enhanced Transformer for sentiment analysis (SentiFormer) to fuse multiple metadata and the corresponding image into a unified framework. Specifically, we first obtain multiple metadata of the image and unify the representations of diverse data. To adaptively learn the appropriate weights for each metadata, we then design an adaptive relevance learning module to highlight more effective information while suppressing weaker ones. Moreover, we further develop a cross-modal fusion module to fuse the adaptively learned representations and make the final prediction. Extensive experiments on three publicly available datasets demonstrate the superiority and rationality of our proposed method.