Image Captioning at Will: A Versatile Scheme for Effectively Injecting Sentiments into Image Descriptions

TL;DR

This paper introduces two simple yet effective models for injecting sentiments into image captions, enabling the generation of emotional descriptions that are more diverse and appealing, outperforming existing methods.

Contribution

The paper presents novel sentiment injection schemes into image captioning models, enhancing emotional expressiveness without compromising semantic accuracy.

Findings

Models outperform state-of-the-art in sentiment captioning

Easy manipulation of generated sentiments for test images

Simpler models achieve better results than previous approaches

Abstract

Automatic image captioning has recently approached human-level performance due to the latest advances in computer vision and natural language understanding. However, most of the current models can only generate plain factual descriptions about the content of a given image. However, for human beings, image caption writing is quite flexible and diverse, where additional language dimensions, such as emotion, humor and language styles, are often incorporated to produce diverse, emotional, or appealing captions. In particular, we are interested in generating sentiment-conveying image descriptions, which has received little attention. The main challenge is how to effectively inject sentiments into the generated captions without altering the semantic matching between the visual content and the generated descriptions. In this work, we propose two different models, which employ different schemes for injecting sentiments into image captions. Compared with the few existing approaches, the proposed models are much simpler and yet more effective. The experimental results show that our model outperform the state-of-the-art models in generating sentimental (i.e., sentiment-bearing) image captions. In addition, we can also easily manipulate the model by assigning different sentiments to the testing image to generate captions with the corresponding sentiments.