Solving Jigsaw Puzzles in the Wild: Human-Guided Reconstruction of Cultural Heritage Fragments

TL;DR

This paper presents a human-guided framework for reassembling complex archaeological fragments, combining automatic algorithms with interactive human input to improve accuracy and scalability in real-world artifact reconstruction.

Contribution

It introduces a novel hybrid human-in-the-loop approach with two interaction strategies, enhancing large-scale cultural heritage puzzle solving beyond traditional methods.

Findings

Outperforms automatic and manual baselines in accuracy

Enables scalable reconstruction for large, ambiguous puzzles

Demonstrates effectiveness on RePAIR benchmark groups

Abstract

Reassembling real-world archaeological artifacts from fragmented pieces poses significant challenges due to erosion, missing regions, irregular shapes, and large-scale ambiguity. Traditional jigsaw puzzle solvers, often designed for clean synthetic scenarios, struggle under these conditions, especially when the number of fragments grows into the thousands, as in the RePAIR benchmark. In this paper, we propose a human-in-the-loop (HIL) puzzle solving framework designed to address the complexity and scale of real-world cultural heritage reconstruction. Our approach integrates an automatic relaxation-labeling solver with interactive human guidance, allowing users to iteratively lock verified placements, correct errors, and guide the system toward semantically and geometrically coherent assemblies. We introduce two complementary interaction strategies, Iterative Anchoring and Continuous Interactive Refinement, which support scalable reconstruction across varying levels of ambiguity and puzzle size. Experiments on several RePAIR groups demonstrate that our hybrid approach substantially outperforms both fully automatic and manual baselines in accuracy and efficiency, offering a practical solution for large-scale expert-in-the-loop artifact reassembly.