From Image Captioning to Visual Storytelling

TL;DR

This paper presents a unified framework for visual storytelling that combines image captioning and narrative generation, improving story quality, training efficiency, and introducing a new metric for evaluating storytelling models.

Contribution

It introduces a novel approach that treats visual storytelling as an extension of image captioning, enhancing coherence and grounding while reducing training time.

Findings

Integrated captioning and storytelling improves story quality.

The framework is faster to train and more reproducible.

The new ideality metric correlates with human-likeness.

Abstract

Visual Storytelling is a challenging multimodal task between Vision & Language, where the purpose is to generate a story for a stream of images. Its difficulty lies on the fact that the story should be both grounded to the image sequence but also narrative and coherent. The aim of this work is to balance between these aspects, by treating Visual Storytelling as a superset of Image Captioning, an approach quite different compared to most of prior relevant studies. This means that we firstly employ a vision-to-language model for obtaining captions of the input images, and then, these captions are transformed into coherent narratives using language-to-language methods. Our multifarious evaluation shows that integrating captioning and storytelling under a unified framework, has a positive impact on the quality of the produced stories. In addition, compared to numerous previous studies, this approach accelerates training time and makes our framework readily reusable and reproducible by anyone interested. Lastly, we propose a new metric/tool, named ideality, that can be used to simulate how far some results are from an oracle model, and we apply it to emulate human-likeness in visual storytelling.