SemStyle: Learning to Generate Stylised Image Captions using Unaligned Text

TL;DR

SemStyle is a novel model that generates visually relevant, styled image captions from unaligned text by separating semantics and style, leveraging large corpora of styled language.

Contribution

It introduces a new approach to stylized caption generation that does not require aligned image-caption pairs, using a semantic representation and a unified language model.

Findings

Captions preserve image semantics and are style shifted.

Automatic and manual evaluations confirm relevance and style adaptation.

Model leverages large unaligned styled text corpora.

Abstract

Linguistic style is an essential part of written communication, with the power to affect both clarity and attractiveness. With recent advances in vision and language, we can start to tackle the problem of generating image captions that are both visually grounded and appropriately styled. Existing approaches either require styled training captions aligned to images or generate captions with low relevance. We develop a model that learns to generate visually relevant styled captions from a large corpus of styled text without aligned images. The core idea of this model, called SemStyle, is to separate semantics and style. One key component is a novel and concise semantic term representation generated using natural language processing techniques and frame semantics. In addition, we develop a unified language model that decodes sentences with diverse word choices and syntax for different styles. Evaluations, both automatic and manual, show captions from SemStyle preserve image semantics, are descriptive, and are style shifted. More broadly, this work provides possibilities to learn richer image descriptions from the plethora of linguistic data available on the web.