Word-Level Fine-Grained Story Visualization
Bowen Li, Thomas Lukasiewicz

TL;DR
This paper introduces a novel approach for story visualization that enhances image quality and consistency by using a new sentence representation and an improved discriminator, outperforming existing methods without extra semantic tools.
Contribution
It proposes a new word-level sentence representation and a fusion feature discriminator with extended spatial attention for better story visualization.
Findings
Superior performance on multiple datasets
No need for segmentation masks or auxiliary captioning
Human evaluations favor the proposed method
Abstract
Story visualization aims to generate a sequence of images to narrate each sentence in a multi-sentence story with a global consistency across dynamic scenes and characters. Current works still struggle with output images' quality and consistency, and rely on additional semantic information or auxiliary captioning networks. To address these challenges, we first introduce a new sentence representation, which incorporates word information from all story sentences to mitigate the inconsistency problem. Then, we propose a new discriminator with fusion features and further extend the spatial attention to improve image quality and story consistency. Extensive experiments on different datasets and human evaluation demonstrate the superior performance of our approach, compared to state-of-the-art methods, neither using segmentation masks nor auxiliary captioning networks.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMultimodal Machine Learning Applications · Video Analysis and Summarization · Human Pose and Action Recognition
