Multimodal Search on Iconclass using Vision-Language Pre-Trained Models

TL;DR

This paper introduces a multimodal search engine for Iconclass that leverages a pre-trained vision-language model, CLIP, enabling semantic exploration through images and text, enhancing retrieval in cultural heritage contexts.

Contribution

It presents a novel application of CLIP for Iconclass, integrating visual and textual modalities for improved semantic search in cultural heritage classification systems.

Findings

Effective retrieval of Iconclass concepts using visual queries.

Enhanced semantic exploration with multimodal search.

Potential for improved cultural heritage information retrieval.

Abstract

Terminology sources, such as controlled vocabularies, thesauri and classification systems, play a key role in digitizing cultural heritage. However, Information Retrieval (IR) systems that allow to query and explore these lexical resources often lack an adequate representation of the semantics behind the user's search, which can be conveyed through multiple expression modalities (e.g., images, keywords or textual descriptions). This paper presents the implementation of a new search engine for one of the most widely used iconography classification system, Iconclass. The novelty of this system is the use of a pre-trained vision-language model, namely CLIP, to retrieve and explore Iconclass concepts using visual or textual queries.