CaptionSmiths: Flexibly Controlling Language Pattern in Image Captioning

TL;DR

CaptionSmiths introduces a flexible image captioning model that smoothly controls language properties like length and descriptiveness by interpolating between learned property representations, outperforming baselines in lexical alignment and property control.

Contribution

It presents a novel method to condition image captioning models on continuous properties without human annotation, enabling smooth property transitions.

Findings

Reduces caption length control error by 506%

Achieves higher lexical alignment than baselines

Enables smooth transition of caption properties

Abstract

An image captioning model flexibly switching its language pattern, e.g., descriptiveness and length, should be useful since it can be applied to diverse applications. However, despite the dramatic improvement in generative vision-language models, fine-grained control over the properties of generated captions is not easy due to two reasons: (i) existing models are not given the properties as a condition during training and (ii) existing models cannot smoothly transition its language pattern from one state to the other. Given this challenge, we propose a new approach, CaptionSmiths, to acquire a single captioning model that can handle diverse language patterns. First, our approach quantifies three properties of each caption, length, descriptiveness, and uniqueness of a word, as continuous scalar values, without human annotation. Given the values, we represent the conditioning via interpolation between two endpoint vectors corresponding to the extreme states, e.g., one for a very short caption and one for a very long caption. Empirical results demonstrate that the resulting model can smoothly change the properties of the output captions and show higher lexical alignment than baselines. For instance, CaptionSmiths reduces the error in controlling caption length by 506\% despite better lexical alignment. Code will be available on https://github.com/omron-sinicx/captionsmiths.