Visual and Textual Sentiment Analysis Using Deep Fusion Convolutional Neural Networks

TL;DR

This paper introduces a deep fusion convolutional neural network that jointly learns from visual and textual data for sentiment analysis, leveraging their semantic correlation to improve prediction accuracy.

Contribution

It proposes an end-to-end deep fusion model that combines visual and textual sentiment features in social media data, outperforming existing single-modality methods.

Findings

Achieves promising results on two benchmark datasets.

Outperforms state-of-the-art methods in sentiment prediction.

Demonstrates effectiveness of multimodal fusion in sentiment analysis.

Abstract

Sentiment analysis is attracting more and more attentions and has become a very hot research topic due to its potential applications in personalized recommendation, opinion mining, etc. Most of the existing methods are based on either textual or visual data and can not achieve satisfactory results, as it is very hard to extract sufficient information from only one single modality data. Inspired by the observation that there exists strong semantic correlation between visual and textual data in social medias, we propose an end-to-end deep fusion convolutional neural network to jointly learn textual and visual sentiment representations from training examples. The two modality information are fused together in a pooling layer and fed into fully-connected layers to predict the sentiment polarity. We evaluate the proposed approach on two widely used data sets. Results show that our method achieves promising result compared with the state-of-the-art methods which clearly demonstrate its competency.