Multimodal Cultural Heritage Knowledge Graph Extension with Language and Vision Models

TL;DR

This paper presents a multimodal knowledge graph extension approach for cultural heritage data, integrating text and images with large language and vision models, demonstrated on the French WJoconde dataset.

Contribution

It introduces a new multimodal cultural heritage knowledge graph, variants for research, a benchmark for knowledge graph completion, and a framework leveraging LLMs and VLMs for KG extension.

Findings

Enhanced KGs with high reliability through multimodal data integration

Built a comprehensive benchmark for KGC on cultural heritage data

Open-sourced code, datasets, and interactive data access

Abstract

The preservation and interpretation of cultural heritage increasingly rely on digital technologies, among which Knowledge Graphs (KGs) stand out for their ability to structure vast amounts of data. However, the construction and expansion of these KGs often face challenges due to the diverse and complex nature of cultural heritage information. In this paper, we propose a novel approach for extending KG resources in the domain of cultural heritage, which we applied to French data. First, we introduce a new knowledge graph in the domain of French cultural heritage, WJoconde, which is distinguished by its multimodality as it integrates both textual and image information of the entities. We further introduce three variants of WJoconde to facilitate downstream research, such as Knowledge Graph Completion (KGC). We also built a comprehensive benchmark for KGC methods on our dataset. Second, we propose a new framework for extending cultural heritage KGs using multi-modal approaches leveraging Large Language Models (LLMs) and Vision-Language Models (VLMs), which includes automated data extraction from unstructured resources combined with a special validation pipeline for grounding the output of both models, to further extend WJoconde. Our results show that by integrating the rich text and image information in cultural heritage data, we can efficiently enhance KGs with high reliability. We open-source all code and benchmark datasets with text and images, as well as the original data with an interactive access point