Visual Storytelling with Question-Answer Plans
Danyang Liu, Mirella Lapata, Frank Keller

TL;DR
This paper introduces a novel visual storytelling framework that combines visual representations, pretrained language models, and planning via question-answer pairs to produce more coherent and engaging narratives from image sequences.
Contribution
The proposed method uniquely integrates question-answer plans with visual and language models, improving story coherence and interest over existing approaches.
Findings
Blueprint-based models outperform baselines in coherence.
Generated stories are more interesting and natural.
Automatic and human evaluations confirm improvements.
Abstract
Visual storytelling aims to generate compelling narratives from image sequences. Existing models often focus on enhancing the representation of the image sequence, e.g., with external knowledge sources or advanced graph structures. Despite recent progress, the stories are often repetitive, illogical, and lacking in detail. To mitigate these issues, we present a novel framework which integrates visual representations with pretrained language models and planning. Our model translates the image sequence into a visual prefix, a sequence of continuous embeddings which language models can interpret. It also leverages a sequence of question-answer pairs as a blueprint plan for selecting salient visual concepts and determining how they should be assembled into a narrative. Automatic and human evaluation on the VIST benchmark (Huang et al., 2016) demonstrates that blueprint-based models generate…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMultimodal Machine Learning Applications · Topic Modeling · Natural Language Processing Techniques
MethodsFocus
