Exploiting CLIP-based Multi-modal Approach for Artwork Classification and Retrieval

TL;DR

This paper explores the application of the CLIP multimodal model to artwork classification and retrieval, demonstrating its strong zero-shot classification performance and promising retrieval results on a large web-crawled art dataset.

Contribution

It is the first comprehensive study applying CLIP to artwork domain tasks, showing its effectiveness in zero-shot classification and retrieval.

Findings

CLIP achieves high zero-shot classification accuracy on artwork images.

The model shows promising results in artwork-to-artwork retrieval.

CLIP outperforms traditional methods on the NoisyArt dataset.

Abstract

Given the recent advances in multimodal image pretraining where visual models trained with semantically dense textual supervision tend to have better generalization capabilities than those trained using categorical attributes or through unsupervised techniques, in this work we investigate how recent CLIP model can be applied in several tasks in artwork domain. We perform exhaustive experiments on the NoisyArt dataset which is a dataset of artwork images crawled from public resources on the web. On such dataset CLIP achieves impressive results on (zero-shot) classification and promising results in both artwork-to-artwork and description-to-artwork domain.