Neural-based Cross-modal Search and Retrieval of Artwork

TL;DR

This paper presents BoonArt, a deep learning-based cross-modal search engine for artwork that significantly improves retrieval accuracy by leveraging visual-semantic embedding networks on the ArtUK dataset.

Contribution

It introduces BoonArt, the first VSE-based system tailored for artwork datasets, enabling effective image-text and text-image retrieval.

Findings

Achieved 97% Recall@10 for image-to-text retrieval

Achieved 97.4% Recall@10 for text-to-image retrieval

Outperforms traditional search engines on ArtUK dataset

Abstract

Creating an intelligent search and retrieval system for artwork images, particularly paintings, is crucial for documenting cultural heritage, fostering wider public engagement, and advancing artistic analysis and interpretation. Visual-Semantic Embedding (VSE) networks are deep learning models used for information retrieval, which learn joint representations of textual and visual data, enabling 1) cross-modal search and retrieval tasks, such as image-to-text and text-to-image retrieval; and 2) relation-focused retrieval to capture entity relationships and provide more contextually relevant search results. Although VSE networks have played a significant role in cross-modal information retrieval, their application to painting datasets, such as ArtUK, remains unexplored. This paper introduces BoonArt, a VSE-based cross-modal search engine that allows users to search for images using textual queries, and to obtain textual descriptions along with the corresponding images when using image queries. The performance of BoonArt was evaluated using the ArtUK dataset. Experimental evaluations revealed that BoonArt achieved 97% Recall@10 for image-to-text retrieval, and 97.4% Recall@10 for text-to-image Retrieval. By bridging the gap between textual and visual modalities, BoonArt provides a much-improved search performance compared to traditional search engines, such as the one provided by the ArtUK website. BoonArt can be utilised to work with other artwork datasets.