Multi-Class Zero-Shot Learning for Artistic Material Recognition

TL;DR

This paper presents a zero-shot learning model that accurately identifies artistic materials from textual descriptions, enabling material recognition on artworks from different museum collections without prior labeled examples.

Contribution

The authors develop a novel zero-shot learning approach for artistic material recognition using textual descriptions, demonstrating transferability across museum datasets.

Findings

Achieved 48.42% accuracy on Tate artworks

Model successfully generalizes to a new museum dataset

Demonstrates effectiveness of ZSL in sparse data scenarios

Abstract

Zero-Shot Learning (ZSL) is an extreme form of transfer learning, where no labelled examples of the data to be classified are provided during the training stage. Instead, ZSL uses additional information learned about the domain, and relies upon transfer learning algorithms to infer knowledge about the missing instances. ZSL approaches are an attractive solution for sparse datasets. Here we outline a model to identify the materials with which a work of art was created, by learning the relationship between English descriptions of the subject of a piece and its composite materials. After experimenting with a range of hyper-parameters, we produce a model which is capable of correctly identifying the materials used on pieces from an entirely distinct museum dataset. This model returned a classification accuracy of 48.42% on 5,000 artworks taken from the Tate collection, which is distinct from the Rijksmuseum network used to create and train our model.