A Progressive Evaluation Framework for Multicultural Analysis of Story Visualization

TL;DR

This paper introduces a comprehensive framework for evaluating the cultural appropriateness of story visualization models across multiple languages, highlighting biases and proposing new metrics and automated assessment methods.

Contribution

It presents a novel Progressive Multicultural Evaluation Framework with five new metrics and an MLLM-based automated assessment, addressing cultural fidelity in story visualization.

Findings

Models perform better on real-world datasets than animated ones.

Cultural biases are evident, with English and Chinese cultures showing different strengths.

Hindi shows lower performance, indicating cultural bias in models.

Abstract

Recent advancements in text-to-image generative models have improved narrative consistency in story visualization. However, current story visualization models often overlook cultural dimensions, resulting in visuals that lack authenticity and cultural fidelity. In this study, we conduct a comprehensive multicultural analysis of story visualization using current text-to-image models across multilingual settings on two datasets: FlintstonesSV and VIST. To assess cultural dimensions rigorously, we propose a Progressive Multicultural Evaluation Framework and introduce five story visualization metrics, Cultural Appropriateness, Visual Aesthetics, Cohesion, Semantic Consistency, and Object Presence, that are not addressed by existing metrics. We further automate assessment through an MLLM-as-Jury framework that approximates human judgment. Human evaluations show that models generate more coherent, visually appealing, and culturally appropriate stories for real-world datasets than for animated ones. The generated stories exhibit a stronger alignment with English-speaking cultures across all metrics except Cohesion, where Chinese performs better. In contrast, Hindi ranks lowest on all metrics except Visual Aesthetics, reflecting real-world cultural biases embedded in current models. This multicultural analysis provides a foundation for future research aimed at generating culturally appropriate and inclusive visual stories across diverse linguistic and cultural settings.