StoryReasoning Dataset: Using Chain-of-Thought for Scene Understanding and Grounded Story Generation

TL;DR

This paper introduces the StoryReasoning dataset and a novel approach for grounded visual storytelling that maintains character consistency and reduces hallucinations through chain-of-thought reasoning and cross-frame visual re-identification.

Contribution

It presents a new dataset with structured scene analysis and grounded stories, and a fine-tuned model that improves scene understanding and story coherence.

Findings

Reduced hallucinations by 12.3% on average.

Improved creativity scores by 31%.

Demonstrated effective cross-frame object re-identification.

Abstract

Visual storytelling systems struggle to maintain character identity across frames and link actions to appropriate subjects, frequently leading to referential hallucinations. These issues can be addressed through grounding of characters, objects, and other entities on the visual elements. We propose StoryReasoning, a dataset containing 4,178 stories derived from 52,016 movie images, with both structured scene analyses and grounded stories. Each story maintains character and object consistency across frames while explicitly modeling multi-frame relationships through structured tabular representations. Our approach features cross-frame object re-identification using visual similarity and face recognition, chain-of-thought reasoning for explicit narrative modeling, and a grounding scheme that links textual elements to visual entities across multiple frames. We establish baseline performance by fine-tuning Qwen2.5-VL 7B, creating Qwen Storyteller, which performs end-to-end object detection, re-identification, and landmark detection while maintaining consistent object references throughout the story. Evaluation demonstrates a reduction from 4.06 to 3.56 (-12.3%) hallucinations on average per story and an improvement in creativity from 2.58 to 3.38 (+31.0%) when compared to a non-fine-tuned model.