Multilingual Visual Sentiment Concept Matching

TL;DR

This paper introduces computational tools for analyzing a large multilingual dataset of visual concepts and images, revealing cultural differences in visual sentiment perception through clustering and semantic analysis.

Contribution

It presents a novel framework for representing, clustering, and analyzing multilingual visual sentiment concepts using crowdsourcing and word embeddings.

Findings

Effective clustering of multilingual visual concepts.

Insights into cultural differences in visual sentiment.

A new evaluation method for semantic relatedness.

Abstract

The impact of culture in visual emotion perception has recently captured the attention of multimedia research. In this study, we pro- vide powerful computational linguistics tools to explore, retrieve and browse a dataset of 16K multilingual affective visual concepts and 7.3M Flickr images. First, we design an effective crowdsourc- ing experiment to collect human judgements of sentiment connected to the visual concepts. We then use word embeddings to repre- sent these concepts in a low dimensional vector space, allowing us to expand the meaning around concepts, and thus enabling insight about commonalities and differences among different languages. We compare a variety of concept representations through a novel evaluation task based on the notion of visual semantic relatedness. Based on these representations, we design clustering schemes to group multilingual visual concepts, and evaluate them with novel metrics based on the crowdsourced sentiment annotations as well as visual semantic relatedness. The proposed clustering framework enables us to analyze the full multilingual dataset in-depth and also show an application on a facial data subset, exploring cultural in- sights of portrait-related affective visual concepts.