A System for Image Understanding using Sensemaking and Narrative

TL;DR

This paper presents a computational system for visual storytelling that leverages sensemaking and narrative theories to enhance understanding and storytelling capabilities in images.

Contribution

It introduces a novel system integrating sensemaking and narrative concepts into visual storytelling, bridging cognitive theories with computational implementation.

Findings

System demonstrates improved coherence in visual stories

Enhances understanding of how narrative structures aid image interpretation

Provides a framework for future AI storytelling applications

Abstract

Sensemaking and narrative are two inherently interconnected concepts about how people understand the world around them. Sensemaking is the process by which people structure and interconnect the information they encounter in the world with the knowledge and inferences they have made in the past. Narratives are important constructs that people use sensemaking to create; ones that reflect provide a more holistic account of the world than the information within any given narrative is able to alone. Both are important to how human beings parse the world, and both would be valuable for a computational system attempting to do the same. In this paper, we discuss theories of sensemaking and narrative with respect to how people build an understanding of the world based on the information they encounter, as well as the links between the fields of sensemaking and narrative research. We highlight a specific computational task, visual storytelling, whose solutions we believe can be enhanced by employing a sensemaking and narrative component. We then describe our system for visual storytelling using sensemaking and narrative and discuss examples from its current implementation.