Re-assembling the past: The RePAIR dataset and benchmark for real world 2D and 3D puzzle solving

TL;DR

This paper introduces the RePAIR dataset, a challenging benchmark with realistic 2D and 3D puzzle fragments from Pompeii, designed to advance computational puzzle-solving methods by providing complex, real-world data.

Contribution

The paper presents the RePAIR dataset with realistic, multi-modal fragments and a benchmark for 2D and 3D puzzle solving, addressing gaps in current synthetic-focused datasets.

Findings

Current methods show significant performance gaps on the dataset.

The dataset includes high-resolution images, 3D scans, and archaeologist annotations.

Realistic, eroded fragments challenge existing puzzle-solving algorithms.

Abstract

This paper proposes the RePAIR dataset that represents a challenging benchmark to test modern computational and data driven methods for puzzle-solving and reassembly tasks. Our dataset has unique properties that are uncommon to current benchmarks for 2D and 3D puzzle solving. The fragments and fractures are realistic, caused by a collapse of a fresco during a World War II bombing at the Pompeii archaeological park. The fragments are also eroded and have missing pieces with irregular shapes and different dimensions, challenging further the reassembly algorithms. The dataset is multi-modal providing high resolution images with characteristic pictorial elements, detailed 3D scans of the fragments and meta-data annotated by the archaeologists. Ground truth has been generated through several years of unceasing fieldwork, including the excavation and cleaning of each fragment, followed by manual puzzle solving by archaeologists of a subset of approx. 1000 pieces among the 16000 available. After digitizing all the fragments in 3D, a benchmark was prepared to challenge current reassembly and puzzle-solving methods that often solve more simplistic synthetic scenarios. The tested baselines show that there clearly exists a gap to fill in solving this computationally complex problem.