Matrix3D: Large Photogrammetry Model All-in-One

TL;DR

Matrix3D is a comprehensive photogrammetry model that unifies pose estimation, depth prediction, and view synthesis using a multi-modal diffusion transformer, leveraging mask learning to train on diverse incomplete data for improved 3D content creation.

Contribution

The paper introduces Matrix3D, a novel unified model employing a multi-modal diffusion transformer and mask learning strategy to perform multiple photogrammetry tasks with incomplete data.

Findings

Achieves state-of-the-art results in pose estimation.

Demonstrates superior novel view synthesis performance.

Enables fine-grained control for 3D content creation.

Abstract

We present Matrix3D, a unified model that performs several photogrammetry subtasks, including pose estimation, depth prediction, and novel view synthesis using just the same model. Matrix3D utilizes a multi-modal diffusion transformer (DiT) to integrate transformations across several modalities, such as images, camera parameters, and depth maps. The key to Matrix3D's large-scale multi-modal training lies in the incorporation of a mask learning strategy. This enables full-modality model training even with partially complete data, such as bi-modality data of image-pose and image-depth pairs, thus significantly increases the pool of available training data. Matrix3D demonstrates state-of-the-art performance in pose estimation and novel view synthesis tasks. Additionally, it offers fine-grained control through multi-round interactions, making it an innovative tool for 3D content creation. Project page: https://nju-3dv.github.io/projects/matrix3d.