Real3D: Scaling Up Large Reconstruction Models with Real-World Images
Hanwen Jiang, Qixing Huang, Georgios Pavlakos

TL;DR
Real3D introduces a novel self-training framework enabling large reconstruction models to be trained effectively using only single-view real-world images, overcoming limitations of synthetic datasets and improving performance across diverse data sources.
Contribution
The paper presents the first large reconstruction model trained with real-world images using a self-training approach with unsupervised pixel- and semantic-level losses.
Findings
Outperforms prior models on multiple evaluation benchmarks.
Effectively leverages real-world images without ground-truth 3D.
Scales to diverse in-the-wild image datasets.
Abstract
The default strategy for training single-view Large Reconstruction Models (LRMs) follows the fully supervised route using large-scale datasets of synthetic 3D assets or multi-view captures. Although these resources simplify the training procedure, they are hard to scale up beyond the existing datasets and they are not necessarily representative of the real distribution of object shapes. To address these limitations, in this paper, we introduce Real3D, the first LRM system that can be trained using single-view real-world images. Real3D introduces a novel self-training framework that can benefit from both the existing synthetic data and diverse single-view real images. We propose two unsupervised losses that allow us to supervise LRMs at the pixel- and semantic-level, even for training examples without ground-truth 3D or novel views. To further improve performance and scale up the image…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMedical Image Segmentation Techniques · Computer Graphics and Visualization Techniques · 3D Shape Modeling and Analysis
