Harnessing Self-Supervised Features for Art Classification

TL;DR

This paper investigates the use of self-supervised and supervised models as feature extractors for art classification and retrieval, demonstrating that self-supervised backbones improve performance and are applicable in real-world VR museum navigation.

Contribution

It systematically evaluates self-supervised models like DINO and CLIP for artwork classification and retrieval, highlighting their effectiveness over traditional methods.

Findings

Self-supervised backbones improve classification accuracy.

Evaluation of DINO and CLIP models across multiple strategies.

Insights into VR applications for museum navigation.

Abstract

Classifying artworks presents a significant challenge due to the complex interplay of fine-grained details and abstract features that condition the style or genre of an artwork. This paper presents a systematic investigation of the effectiveness of supervised and self-supervised backbones as feature extractors for both artwork classification and retrieval, with a particular focus on paintings. We conduct an extensive experimental evaluation using the DINO family and CLIP models, assessing multiple classification strategies and feature representations. Our results demonstrate that employing a self-supervised backbone leads to consistent improvements in artwork classification performance. Moreover, our work provides insights into the applicability of classification and retrieval modules in real-world applications, such as virtual reality (VR) applications that support museum navigation.