Ordered Attention for Coherent Visual Storytelling

TL;DR

This paper introduces ordered image attention (OIA) and image-sentence attention (ISA) mechanisms to generate more coherent, focused, and image-grounded stories from image sequences, improving METEOR scores and human judgments.

Contribution

It proposes novel ordered attention mechanisms and an adaptive prior to enhance visual storytelling coherence and reduce linguistic errors.

Findings

METEOR score improved by 1% on VIST dataset

Human study shows increased story coherency and focus

Generated stories are more shareable and image-grounded

Abstract

We address the problem of visual storytelling, i.e., generating a story for a given sequence of images. While each sentence of the story should describe a corresponding image, a coherent story also needs to be consistent and relate to both future and past images. To achieve this we develop ordered image attention (OIA). OIA models interactions between the sentence-corresponding image and important regions in other images of the sequence. To highlight the important objects, a message-passing-like algorithm collects representations of those objects in an order-aware manner. To generate the story's sentences, we then highlight important image attention vectors with an Image-Sentence Attention (ISA). Further, to alleviate common linguistic mistakes like repetitiveness, we introduce an adaptive prior. The obtained results improve the METEOR score on the VIST dataset by 1%. In addition, an extensive human study verifies coherency improvements and shows that OIA and ISA generated stories are more focused, shareable, and image-grounded.