Multi-Modal Semantic Parsing for the Interpretation of Tombstone Inscriptions

TL;DR

This paper presents a multi-modal framework utilizing vision-language models and retrieval-augmented generation to interpret and organize tombstone inscriptions, significantly improving accuracy over traditional OCR methods and aiding heritage preservation.

Contribution

It introduces the first formal approach to tombstone understanding using large vision-language models combined with retrieval-augmented generation for semantic parsing.

Findings

Parsing accuracy improved from 36.1 to 89.5 F1 score.

Model robust across diverse linguistic and cultural inscriptions.

Effective under simulated physical degradation conditions.

Abstract

Tombstones are historically and culturally rich artifacts, encapsulating individual lives, community memory, historical narratives and artistic expression. Yet, many tombstones today face significant preservation challenges, including physical erosion, vandalism, environmental degradation, and political shifts. In this paper, we introduce a novel multi-modal framework for tombstones digitization, aiming to improve the interpretation, organization and retrieval of tombstone content. Our approach leverages vision-language models (VLMs) to translate tombstone images into structured Tombstone Meaning Representations (TMRs), capturing both image and text information. To further enrich semantic parsing, we incorporate retrieval-augmented generation (RAG) for integrate externally dependent elements such as toponyms, occupation codes, and ontological concepts. Compared to traditional OCR-based pipelines, our method improves parsing accuracy from an F1 score of 36.1 to 89.5. We additionally evaluate the model's robustness across diverse linguistic and cultural inscriptions, and simulate physical degradation through image fusion to assess performance under noisy or damaged conditions. Our work represents the first attempt to formalize tombstone understanding using large vision-language models, presenting implications for heritage preservation.