Deepzzle: Solving Visual Jigsaw Puzzles with Deep Learning andShortest Path Optimization

TL;DR

Deepzzle introduces a novel deep learning and shortest path optimization approach to reassemble images from fragmented pieces with gaps, mimicking archaeological erosion, and handles complex reassembly challenges with new metrics and graph analysis.

Contribution

The paper presents a two-step method combining neural network predictions and graph optimization for challenging image reassembly tasks, including the effect of branch-cut analysis.

Findings

Effective reassembly of complex images with gaps and extraneous pieces.

Comparison shows superiority over existing methods.

New dataset and metric tailored for fragmented image reassembly.

Abstract

We tackle the image reassembly problem with wide space between the fragments, in such a way that the patterns and colors continuity is mostly unusable. The spacing emulates the erosion of which the archaeological fragments suffer. We crop-square the fragments borders to compel our algorithm to learn from the content of the fragments. We also complicate the image reassembly by removing fragments and adding pieces from other sources. We use a two-step method to obtain the reassemblies: 1) a neural network predicts the positions of the fragments despite the gaps between them; 2) a graph that leads to the best reassemblies is made from these predictions. In this paper, we notably investigate the effect of branch-cut in the graph of reassemblies. We also provide a comparison with the literature, solve complex images reassemblies, explore at length the dataset, and propose a new metric that suits its specificities. Keywords: image reassembly, jigsaw puzzle, deep learning, graph, branch-cut, cultural heritage