Multi-View 3D Reconstruction using Knowledge Distillation

TL;DR

This paper introduces a knowledge distillation approach to develop efficient 3D reconstruction models, using Dust3r as a teacher, with Vision Transformers showing superior performance over CNNs.

Contribution

It proposes a novel student-teacher framework for 3D reconstruction, exploring different architectures and training strategies to replicate Dust3r's outputs efficiently.

Findings

Vision Transformer outperforms CNN in 3D reconstruction quality

Pre-trained models improve performance over models trained from scratch

Ablation studies identify key hyperparameters for optimal results

Abstract

Large Foundation Models like Dust3r can produce high quality outputs such as pointmaps, camera intrinsics, and depth estimation, given stereo-image pairs as input. However, the application of these outputs on tasks like Visual Localization requires a large amount of inference time and compute resources. To address these limitations, in this paper, we propose the use of a knowledge distillation pipeline, where we aim to build a student-teacher model with Dust3r as the teacher and explore multiple architectures of student models that are trained using the 3D reconstructed points output by Dust3r. Our goal is to build student models that can learn scene-specific representations and output 3D points with replicable performance such as Dust3r. The data set we used to train our models is 12Scenes. We test two main architectures of models: a CNN-based architecture and a Vision Transformer based architecture. For each architecture, we also compare the use of pre-trained models against models built from scratch. We qualitatively compare the reconstructed 3D points output by the student model against Dust3r's and discuss the various features learned by the student model. We also perform ablation studies on the models through hyperparameter tuning. Overall, we observe that the Vision Transformer presents the best performance visually and quantitatively.