From Pixels to Sentiment: Fine-tuning CNNs for Visual Sentiment Prediction

TL;DR

This paper investigates how fine-tuning CNNs can improve automated visual sentiment analysis, demonstrating accuracy gains and providing insights into the learned visual features associated with sentiment in social media images.

Contribution

It introduces specific CNN fine-tuning techniques and architecture modifications that enhance sentiment prediction accuracy on social media image datasets.

Findings

Accuracy improvements over prior art achieved

Visualizations reveal learned patterns linked to sentiment

Fine-tuning strategies enhance model interpretability

Abstract

Visual multimedia have become an inseparable part of our digital social lives, and they often capture moments tied with deep affections. Automated visual sentiment analysis tools can provide a means of extracting the rich feelings and latent dispositions embedded in these media. In this work, we explore how Convolutional Neural Networks (CNNs), a now de facto computational machine learning tool particularly in the area of Computer Vision, can be specifically applied to the task of visual sentiment prediction. We accomplish this through fine-tuning experiments using a state-of-the-art CNN and via rigorous architecture analysis, we present several modifications that lead to accuracy improvements over prior art on a dataset of images from a popular social media platform. We additionally present visualizations of local patterns that the network learned to associate with image sentiment for insight into how visual positivity (or negativity) is perceived by the model.