Towards Scalable Spatial Intelligence via 2D-to-3D Data Lifting

TL;DR

This paper introduces a scalable pipeline that converts single-view images into realistic 3D representations, significantly reducing data collection costs and enhancing spatial scene understanding for AI systems.

Contribution

The authors present a novel method for automatic, scale-aware 3D data generation from 2D images, bridging the gap between abundant imagery and 3D spatial intelligence needs.

Findings

Generated datasets improve 3D perception tasks

Method reduces data collection costs

Enhanced spatial reasoning in AI systems

Abstract

Spatial intelligence is emerging as a transformative frontier in AI, yet it remains constrained by the scarcity of large-scale 3D datasets. Unlike the abundant 2D imagery, acquiring 3D data typically requires specialized sensors and laborious annotation. In this work, we present a scalable pipeline that converts single-view images into comprehensive, scale- and appearance-realistic 3D representations - including point clouds, camera poses, depth maps, and pseudo-RGBD - via integrated depth estimation, camera calibration, and scale calibration. Our method bridges the gap between the vast repository of imagery and the increasing demand for spatial scene understanding. By automatically generating authentic, scale-aware 3D data from images, we significantly reduce data collection costs and open new avenues for advancing spatial intelligence. We release two generated spatial datasets, i.e., COCO-3D and Objects365-v2-3D, and demonstrate through extensive experiments that our generated data can benefit various 3D tasks, ranging from fundamental perception to MLLM-based reasoning. These results validate our pipeline as an effective solution for developing AI systems capable of perceiving, understanding, and interacting with physical environments.