I Prompt, it Generates, we Negotiate. Exploring Text-Image Intertextuality in Human-AI Co-Creation of Visual Narratives with VLMs

TL;DR

This study explores how humans and AI collaboratively create visual narratives using VLMs, revealing strategies, collaboration patterns, and challenges in understanding and leveraging text-image intertextuality.

Contribution

It provides an empirical analysis of text-image intertextuality in human-AI co-creation and proposes design implications for role-based AI storytelling assistants.

Findings

Users develop strategies to harness AI's semantic surplus.

Four collaboration patterns identified: Educational, Technical, Visual.

Challenges include cultural gaps and visual consistency issues.

Abstract

Creating meaningful visual narratives through human-AI collaboration requires understanding how text-image intertextuality emerges when textual intentions meet AI-generated visuals. We conducted a three-phase qualitative study with 15 participants using GPT-4o to investigate how novices navigate sequential visual narratives. Our findings show that users develop strategies to harness AI's semantic surplus by recognizing meaningful visual content beyond literal descriptions, iteratively refining prompts, and constructing narrative significance through complementary text-image relationships. We identified four distinct collaboration patterns and, through fsQCA's analysis, discovered three pathways to successful intertextual collaboration: Educational Collaborator, Technical Expert, and Visual Thinker. However, participants faced challenges, including cultural representation gaps, visual consistency issues, and difficulties translating narrative concepts into visual prompts. These findings contribute to HCI research by providing an empirical account of \textit{text-image intertextuality} in human-AI co-creation and proposing design implications for role-based AI assistants that better support iterative, human-led creative processes in visual storytelling.