Machine-in-the-Loop Rewriting for Creative Image Captioning

TL;DR

This paper introduces a rewriting model that collaborates with humans to enhance creative image captions by locally modifying drafts, resulting in more descriptive and figurative captions, and outperforms baseline models in user studies.

Contribution

The paper presents a novel rewriting approach that allows humans to retain control while improving creative image captions through targeted text modifications.

Findings

The model is rated more helpful than baseline infilling models.

Users produce more descriptive captions with the model.

Collaborative captions are more figurative and expressive.

Abstract

Machine-in-the-loop writing aims to enable humans to collaborate with models to complete their writing tasks more effectively. Prior work has found that providing humans a machine-written draft or sentence-level continuations has limited success since the generated text tends to deviate from humans' intention. To allow the user to retain control over the content, we train a rewriting model that, when prompted, modifies specified spans of text within the user's original draft to introduce descriptive and figurative elements locally in the text. We evaluate the model on its ability to collaborate with humans on the task of creative image captioning. On a user study through Amazon Mechanical Turk, our model is rated to be more helpful than a baseline infilling language model. In addition, third-party evaluation shows that users write more descriptive and figurative captions when collaborating with our model compared to completing the task alone.