Large-scale Classification of Fine-Art Paintings: Learning The Right Metric on The Right Feature

TL;DR

This paper explores the extraction of visual features and the learning of an optimized similarity metric to improve the classification and retrieval of fine-art paintings, aiding multimedia systems in art collection management.

Contribution

It introduces a comprehensive analysis of visual features and metric learning methods tailored for fine-art paintings, enhancing aesthetic and semantic classification tasks.

Findings

Optimized similarity measures improve style and genre prediction.

Feature and metric learning approaches enhance artwork retrieval.

Semantic-level judgments are feasible with learned metrics.

Abstract

In the past few years, the number of fine-art collections that are digitized and publicly available has been growing rapidly. With the availability of such large collections of digitized artworks comes the need to develop multimedia systems to archive and retrieve this pool of data. Measuring the visual similarity between artistic items is an essential step for such multimedia systems, which can benefit more high-level multimedia tasks. In order to model this similarity between paintings, we should extract the appropriate visual features for paintings and find out the best approach to learn the similarity metric based on these features. We investigate a comprehensive list of visual features and metric learning approaches to learn an optimized similarity measure between paintings. We develop a machine that is able to make aesthetic-related semantic-level judgments, such as predicting a painting's style, genre, and artist, as well as providing similarity measures optimized based on the knowledge available in the domain of art historical interpretation. Our experiments show the value of using this similarity measure for the aforementioned prediction tasks.