JRM: Joint Reconstruction Model for Multiple Objects without Alignment

TL;DR

JRM is a 3D generative model that implicitly aggregates multiple unaligned observations to improve object reconstruction, especially in non-rigid scenarios, without explicit alignment.

Contribution

It introduces a novel flow-matching generative approach that handles unaligned data and non-rigid changes, surpassing prior explicit alignment methods.

Findings

JRM outperforms baseline methods in reconstruction quality.

Implicit aggregation enhances robustness to alignment errors.

Handles non-rigid transformations like articulation effectively.

Abstract

Object-centric reconstruction seeks to recover the 3D structure of a scene through composition of independent objects. While this independence can simplify modeling, it discards strong signals that could improve reconstruction, notably repetition where the same object model is seen multiple times in a scene, or across scans. We propose the Joint Reconstruction Model (JRM) to leverage repetition by framing object reconstruction as one of personalized generation: multiple observations share a common subject that should be consistent for all observations, while still adhering to the specific pose and state from each. Prior methods in this direction rely on explicit matching and rigid alignment across observations, making them sensitive to errors and difficult to extend to non-rigid transformations. In contrast, JRM is a 3D flow-matching generative model that implicitly aggregates unaligned observations in its latent space, learning to produce consistent and faithful reconstructions in a data-driven manner without explicit constraints. Evaluations on synthetic and real-world data show that JRM's implicit aggregation removes the need for explicit alignment, improves robustness to incorrect associations, and naturally handles non-rigid changes such as articulation. Overall, JRM outperforms both independent and alignment-based baselines in reconstruction quality.