Deep Models for Visual Sentiment Analysis of Disaster-related Multimedia Content

TL;DR

This paper develops deep learning solutions using Inception-v3 and VggNet-19 models to classify sentiments in disaster-related images shared on social media, addressing a MediaEval 2021 challenge.

Contribution

It introduces fine-tuned deep models for multi-task visual sentiment analysis of disaster images, demonstrating effective performance on multiple classification subtasks.

Findings

Inception-v3 achieved weighted F1-scores around 0.54-0.57 across tasks.

VggNet-19 achieved weighted F1-scores around 0.49-0.58 across tasks.

The proposed models showed encouraging results on all three sentiment classification tasks.

Abstract

This paper presents a solutions for the MediaEval 2021 task namely "Visual Sentiment Analysis: A Natural Disaster Use-case". The task aims to extract and classify sentiments perceived by viewers and the emotional message conveyed by natural disaster-related images shared on social media. The task is composed of three sub-tasks including, one single label multi-class image classification task, and, two multi-label multi-class image classification tasks, with different sets of labels. In our proposed solutions, we rely mainly on two different state-of-the-art models namely, Inception-v3 and VggNet-19, pre-trained on ImageNet, which are fine-tuned for each of the three task using different strategies. Overall encouraging results are obtained on all the three tasks. On the single-label classification task (i.e. Task 1), we obtained the weighted average F1-scores of 0.540 and 0.526 for the Inception-v3 and VggNet-19 based solutions, respectively. On the multi-label classification i.e., Task 2 and Task 3, the weighted F1-score of our Inception-v3 based solutions was 0.572 and 0.516, respectively. Similarly, the weighted F1-score of our VggNet-19 based solution on Task 2 and Task 3 was 0.584 and 0.495, respectively.