E-M3RF: An Equivariant Multimodal 3D Re-assembly Framework

TL;DR

E-M3RF introduces an equivariant multimodal framework for 3D reassembly that leverages geometric and color features, improving accuracy in fragment assembly tasks, especially when geometry alone is ambiguous.

Contribution

The paper presents a novel multimodal 3D reassembly framework combining geometric and color features with equivariant encoding, addressing limitations of prior geometry-only methods.

Findings

Reduces rotation error by 23.1% on RePAIR dataset

Decreases translation error by 13.2%

Achieves 18.4% lower Chamfer Distance compared to competitors

Abstract

3D reassembly is a fundamental geometric problem, and in recent years it has increasingly been challenged by deep learning methods rather than classical optimization. While learning approaches have shown promising results, most still rely primarily on geometric features to assemble a whole from its parts. As a result, methods struggle when geometry alone is insufficient or ambiguous, for example, for small, eroded, or symmetric fragments. Additionally, solutions do not impose physical constraints that explicitly prevent overlapping assemblies. To address these limitations, we introduce E-M3RF, an equivariant multimodal 3D reassembly framework that takes as input the point clouds, containing both point positions and colors of fractured fragments, and predicts the transformations required to reassemble them using SE(3) flow matching. Each fragment is represented by both geometric and color features: i) 3D point positions are encoded as rotationconsistent geometric features using a rotation-equivariant encoder, ii) the colors at each 3D point are encoded with a transformer. The two feature sets are then combined to form a multimodal representation. We experimented on four datasets: two synthetic datasets, Breaking Bad and Fantastic Breaks, and two real-world cultural heritage datasets, RePAIR and Presious, demonstrating that E-M3RF on the RePAIR dataset reduces rotation error by 23.1% and translation error by 13.2%, while Chamfer Distance decreases by 18.4% compared to competing methods.