ReassembleNet: Learnable Keypoints and Diffusion for 2D Fresco Reconstruction

TL;DR

ReassembleNet is a novel deep learning approach that uses keypoints, graph neural networks, and diffusion techniques to improve 2D structure reassembly in complex, real-world scenarios, achieving significant accuracy gains.

Contribution

The paper introduces ReassembleNet, which reduces complexity and enhances multimodal reassembly by learning keypoints and applying diffusion-based pose estimation.

Findings

Improves reassembly accuracy by 57% in rotation and 87% in translation.

Effectively integrates geometric and texture data from multiple modalities.

Reduces computational complexity compared to prior methods.

Abstract

The task of reassembly is a significant challenge across multiple domains, including archaeology, genomics, and molecular docking, requiring the precise placement and orientation of elements to reconstruct an original structure. In this work, we address key limitations in state-of-the-art Deep Learning methods for reassembly, namely i) scalability; ii) multimodality; and iii) real-world applicability: beyond square or simple geometric shapes, realistic and complex erosion, or other real-world problems. We propose ReassembleNet, a method that reduces complexity by representing each input piece as a set of contour keypoints and learning to select the most informative ones by Graph Neural Networks pooling inspired techniques. ReassembleNet effectively lowers computational complexity while enabling the integration of features from multiple modalities, including both geometric and texture data. Further enhanced through pretraining on a semi-synthetic dataset. We then apply diffusion-based pose estimation to recover the original structure. We improve on prior methods by 57% and 87% for RMSE Rotation and Translation, respectively.