A Dataset for Named Entity Recognition and Relation Extraction from Art-historical Image Descriptions

TL;DR

This paper presents FRAME, a comprehensive, annotated dataset of art-historical image descriptions designed to advance Named Entity Recognition and Relation Extraction tasks, supporting knowledge graph development and LLM fine-tuning.

Contribution

The creation of a detailed, multi-layered dataset with extensive annotations aligned to Wikidata, enabling improved NER, RE, and knowledge graph applications in art history.

Findings

Dataset includes 37 entity types with explicit annotations.

Supports benchmarking and fine-tuning of NER and RE systems.

Facilitates zero- and few-shot learning with Large Language Models.

Abstract

This paper introduces FRAME (Fine-grained Recognition of Art-historical Metadata and Entities), a manually annotated dataset of art-historical image descriptions for Named Entity Recognition (NER) and Relation Extraction (RE). Descriptions were collected from museum catalogs, auction listings, open-access platforms, and scholarly databases, then filtered to ensure that each text focuses on a single artwork and contains explicit statements about its material, composition, or iconography. FRAME provides stand-off annotations in three layers: a metadata layer for object-level properties, a content layer for depicted subjects and motifs, and a co-reference layer linking repeated mentions. Across layers, entity spans are labeled with 37 types and connected by typed RE links between mentions. Entity types are aligned with Wikidata to support Named Entity Linking (NEL) and downstream knowledge-graph construction. The dataset is released as UIMA XMI Common Analysis Structure (CAS) files with accompanying images and bibliographic metadata, and can be used to benchmark and fine-tune NER and RE systems, including zero- and few-shot setups with Large Language Models (LLMs).