Context-aware Visual Storytelling with Visual Prefix Tuning and Contrastive Learning

TL;DR

This paper presents a framework for visual storytelling that uses pretrained foundation models with a lightweight mapping network and contrastive learning to generate diverse, coherent, and informative stories from image sequences.

Contribution

It introduces a novel context-aware visual storytelling approach leveraging visual prefix tuning and contrastive learning with minimal training of a lightweight network.

Findings

Generated stories are diverse, coherent, and informative.

The framework outperforms existing methods on automatic and human evaluations.

Contrastive learning improves visual relevance and story quality.

Abstract

Visual storytelling systems generate multi-sentence stories from image sequences. In this task, capturing contextual information and bridging visual variation bring additional challenges. We propose a simple yet effective framework that leverages the generalization capabilities of pretrained foundation models, only training a lightweight vision-language mapping network to connect modalities, while incorporating context to enhance coherence. We introduce a multimodal contrastive objective that also improves visual relevance and story informativeness. Extensive experimental results, across both automatic metrics and human evaluations, demonstrate that the stories generated by our framework are diverse, coherent, informative, and interesting.