ArtContext: Contextualizing Artworks with Open-Access Art History Articles and Wikidata Knowledge through a LoRA-Tuned CLIP Model

TL;DR

ArtContext introduces a pipeline that combines open-access art history articles and Wikidata with a LoRA-tuned CLIP model to effectively annotate artworks with contextual information, enhancing understanding and research in art history.

Contribution

The paper presents a novel corpus collection pipeline and a domain-specific CLIP model, PaintingCLIP, trained with LoRA for improved contextual annotation of artworks.

Findings

PaintingCLIP outperforms standard CLIP in contextual annotation tasks.

The pipeline is adaptable to various humanities domains.

The approach enhances linking artworks with detailed textual context.

Abstract

Many Art History articles discuss artworks in general as well as specific parts of works, such as layout, iconography, or material culture. However, when viewing an artwork, it is not trivial to identify what different articles have said about the piece. Therefore, we propose ArtContext, a pipeline for taking a corpus of Open-Access Art History articles and Wikidata Knowledge and annotating Artworks with this information. We do this using a novel corpus collection pipeline, then learn a bespoke CLIP model adapted using Low-Rank Adaptation (LoRA) to make it domain-specific. We show that the new model, PaintingCLIP, which is weakly supervised by the collected corpus, outperforms CLIP and provides context for a given artwork. The proposed pipeline is generalisable and can be readily applied to numerous humanities areas.