Let Storytelling Tell Vivid Stories: An Expressive and Fluent Multimodal Storyteller

TL;DR

This paper introduces LLaMS, a novel multimodal storytelling pipeline that leverages large language models and a new architecture to generate vivid, consistent, and expressive stories from image streams, outperforming previous methods.

Contribution

The paper presents LLaMS, a new multimodal storytelling framework that enhances factual content and sequence consistency using novel modules and strategies, achieving state-of-the-art performance.

Findings

LLaMS achieves 86% correlation with human judgments.

LLaMS attains 100% story consistency in evaluations.

Ablation studies confirm the effectiveness of proposed modules.

Abstract

Storytelling aims to generate reasonable and vivid narratives based on an ordered image stream. The fidelity to the image story theme and the divergence of story plots attract readers to keep reading. Previous works iteratively improved the alignment of multiple modalities but ultimately resulted in the generation of simplistic storylines for image streams. In this work, we propose a new pipeline, termed LLaMS, to generate multimodal human-level stories that are embodied in expressiveness and consistency. Specifically, by fully exploiting the commonsense knowledge within the LLM, we first employ a sequence data auto-enhancement strategy to enhance factual content expression and leverage a textual reasoning architecture for expressive story generation and prediction. Secondly, we propose SQ-Adatpter module for story illustration generation which can maintain sequence consistency. Numerical results are conducted through human evaluation to verify the superiority of proposed LLaMS. Evaluations show that LLaMS achieves state-of-the-art storytelling performance and 86% correlation and 100% consistency win rate as compared with previous SOTA methods. Furthermore, ablation experiments are conducted to verify the effectiveness of proposed sequence data enhancement and SQ-Adapter.