Moving Pictures of Thought: Extracting Visual Knowledge in Charles S. Peirce's Manuscripts with Vision-Language Models

TL;DR

This paper explores using vision-language models to analyze Charles Peirce's manuscripts, focusing on extracting and interpreting diagrams to enhance understanding of visual reasoning in historical texts.

Contribution

It introduces a workflow combining layout segmentation, VLM analysis, and semiotic prompts to extract and represent diagrammatic knowledge from complex manuscripts.

Findings

VLMs can identify diagrams in heterogeneous manuscript pages

Generated captions facilitate structured knowledge representation

Workflow supports digital analysis of visual artifacts in scholarly texts

Abstract

Diagrams are crucial yet underexplored tools in many disciplines, demonstrating the close connection between visual representation and scholarly reasoning. However, their iconic form poses obstacles to visual studies, intermedial analysis, and text-based digital workflows. In particular, Charles S. Peirce consistently advocated the use of diagrams as essential for reasoning and explanation. His manuscripts, often combining textual content with complex visual artifacts, provide a challenging case for studying documents involving heterogeneous materials. In this preliminary study, we investigate whether Visual Language Models (VLMs) can effectively help us identify and interpret such hybrid pages in context. First, we propose a workflow that (i) segments manuscript page layouts, (ii) reconnects each segment to IIIF-compliant annotations, and (iii) submits fragments containing diagrams to a VLM. In addition, by adopting Peirce's semiotic framework, we designed prompts to extract key knowledge about diagrams and produce concise captions. Finally, we integrated these captions into knowledge graphs, enabling structured representations of diagrammatic content within composite sources.