Automated Mosaic Tesserae Segmentation via Deep Learning Techniques

TL;DR

This paper introduces a deep learning-based method using SAM 2 to automatically segment mosaic tesserae, significantly improving accuracy and paving the way for digital preservation of fragile cultural heritage artifacts.

Contribution

It develops a fine-tuned SAM 2 model and creates an annotated mosaic dataset, achieving higher segmentation accuracy than previous methods.

Findings

Intersection over Union increased from 89.00% to 91.02%

Recall improved from 92.12% to 95.89%

F-measure 3% higher than prior approaches

Abstract

Art is widely recognized as a reflection of civilization and mosaics represent an important part of cultural heritage. Mosaics are an ancient art form created by arranging small pieces, called tesserae, on a surface using adhesive. Due to their age and fragility, they are prone to damage, highlighting the need for digital preservation. This paper addresses the problem of digitizing mosaics by segmenting the tesserae to separate them from the background within the broader field of Image Segmentation in Computer Vision. We propose a method leveraging Segment Anything Model 2 (SAM 2) by Meta AI, a foundation model that outperforms most conventional segmentation models, to automatically segment mosaics. Due to the limited open datasets in the field, we also create an annotated dataset of mosaic images to fine-tune and evaluate the model. Quantitative evaluation on our testing dataset shows notable improvements compared to the baseline SAM 2 model, with Intersection over Union increasing from 89.00% to 91.02% and Recall from 92.12% to 95.89%. Additionally, on a benchmark proposed by a prior approach, our model achieves an F-measure 3% higher than previous methods and reduces the error in the absolute difference between predicted and actual tesserae from 0.20 to just 0.02. The notable performance of the fine-tuned SAM 2 model together with the newly annotated dataset can pave the way for real-time segmentation of mosaic images.