Which Emoji Talks Best for My Picture?

TL;DR

This paper explores how combining visual, textual, and domain knowledge improves emoji recommendation accuracy for multimedia social media posts, outperforming current methods by 9.6%.

Contribution

It introduces a novel emoji recommendation approach that integrates image, text, and domain knowledge, enhancing accuracy over existing models.

Findings

Outperforms state-of-the-art by 9.6% in accuracy.

Utilizes pre-trained classifiers and embeddings on Twitter data.

Validated through user study on MSCOCO images.

Abstract

Emojis have evolved as complementary sources for expressing emotion in social-media platforms where posts are mostly composed of texts and images. In order to increase the expressiveness of the social media posts, users associate relevant emojis with their posts. Incorporating domain knowledge has improved machine understanding of text. In this paper, we investigate whether domain knowledge for emoji can improve the accuracy of emoji recommendation task in case of multimedia posts composed of image and text. Our emoji recommendation can suggest accurate emojis by exploiting both visual and textual content from social media posts as well as domain knowledge from Emojinet. Experimental results using pre-trained image classifiers and pre-trained word embedding models on Twitter dataset show that our results outperform the current state-of-the-art by 9.6\%. We also present a user study evaluation of our recommendation system on a set of images chosen from MSCOCO dataset.