Incorporating Textual Evidence in Visual Storytelling

TL;DR

This paper introduces a novel approach to visual storytelling that incorporates textual evidence from similar images using a two-step ranking and an extended Seq2Seq model, improving story coherence and quality.

Contribution

It presents a new method combining image ranking and a two-channel encoder with attention in Seq2Seq for enhanced visual storytelling.

Findings

Outperforms state-of-the-art models on VIST dataset

Utilizes textual evidence to improve story coherence

Employs a two-step image ranking method

Abstract

Previous work on visual storytelling mainly focused on exploring image sequence as evidence for storytelling and neglected textual evidence for guiding story generation. Motivated by human storytelling process which recalls stories for familiar images, we exploit textual evidence from similar images to help generate coherent and meaningful stories. To pick the images which may provide textual experience, we propose a two-step ranking method based on image object recognition techniques. To utilize textual information, we design an extended Seq2Seq model with two-channel encoder and attention. Experiments on the VIST dataset show that our method outperforms state-of-the-art baseline models without heavy engineering.