Similar Scenes arouse Similar Emotions: Parallel Data Augmentation for Stylized Image Captioning

TL;DR

This paper introduces a novel data augmentation framework for stylized image captioning that leverages similar scenes and emotion extraction to improve caption quality and stylistic consistency, addressing data scarcity issues.

Contribution

It proposes an Extract-Retrieve-Generate framework that enhances stylized captioning by grafting style phrases from similar scenes, significantly improving model performance.

Findings

Boosts caption relevance and style consistency

Alleviates data scarcity in stylized captioning

Outperforms state-of-the-art methods

Abstract

Stylized image captioning systems aim to generate a caption not only semantically related to a given image but also consistent with a given style description. One of the biggest challenges with this task is the lack of sufficient paired stylized data. Many studies focus on unsupervised approaches, without considering from the perspective of data augmentation. We begin with the observation that people may recall similar emotions when they are in similar scenes, and often express similar emotions with similar style phrases, which underpins our data augmentation idea. In this paper, we propose a novel Extract-Retrieve-Generate data augmentation framework to extract style phrases from small-scale stylized sentences and graft them to large-scale factual captions. First, we design the emotional signal extractor to extract style phrases from small-scale stylized sentences. Second, we construct the plugable multi-modal scene retriever to retrieve scenes represented with pairs of an image and its stylized caption, which are similar to the query image or caption in the large-scale factual data. In the end, based on the style phrases of similar scenes and the factual description of the current scene, we build the emotion-aware caption generator to generate fluent and diversified stylized captions for the current scene. Extensive experimental results show that our framework can alleviate the data scarcity problem effectively. It also significantly boosts the performance of several existing image captioning models in both supervised and unsupervised settings, which outperforms the state-of-the-art stylized image captioning methods in terms of both sentence relevance and stylishness by a substantial margin.