ARTeFACT: Benchmarking Segmentation Models on Diverse Analogue Media Damage

TL;DR

ARTeFACT is a comprehensive dataset and benchmark for damage detection in diverse analogue media, highlighting the limitations of current models in generalizing across media types and aiding future research in cultural heritage preservation.

Contribution

We introduce ARTeFACT, a large annotated dataset with textual descriptions for damage detection in various analogue media, and evaluate multiple models revealing their generalization challenges.

Findings

Models struggle to generalize damage detection across media types.

The dataset enables benchmarking of damage detection methods.

Current models have significant limitations in zero-shot and cross-media settings.

Abstract

Accurately detecting and classifying damage in analogue media such as paintings, photographs, textiles, mosaics, and frescoes is essential for cultural heritage preservation. While machine learning models excel in correcting degradation if the damage operator is known a priori, we show that they fail to robustly predict where the damage is even after supervised training; thus, reliable damage detection remains a challenge. Motivated by this, we introduce ARTeFACT, a dataset for damage detection in diverse types analogue media, with over 11,000 annotations covering 15 kinds of damage across various subjects, media, and historical provenance. Furthermore, we contribute human-verified text prompts describing the semantic contents of the images, and derive additional textual descriptions of the annotated damage. We evaluate CNN, Transformer, diffusion-based segmentation models, and foundation vision models in zero-shot, supervised, unsupervised and text-guided settings, revealing their limitations in generalising across media types. Our dataset is available at $\href{https://daniela997.github.io/ARTeFACT/}{https://daniela997.github.io/ARTeFACT/}$ as the first-of-its-kind benchmark for analogue media damage detection and restoration.