Muskie: Multi-view Masked Image Modeling for 3D Vision Pre-training

TL;DR

Muskie introduces a multi-view vision backbone for 3D tasks that learns view-invariant features through a novel pre-training method, significantly improving multi-view correspondence and downstream 3D performance without 3D supervision.

Contribution

Muskie is the first multi-view masked image modeling backbone that processes multiple views simultaneously and enforces multi-view consistency during pre-training.

Findings

Outperforms frame-wise models like DINO in multi-view correspondence accuracy.

Enhances downstream 3D tasks such as camera pose estimation and pointmap reconstruction.

Learns geometric understanding without explicit 3D supervision.

Abstract

We present Muskie, a native multi-view vision backbone designed for 3D vision tasks. Unlike existing models, which are frame-wise and exhibit limited multi-view consistency, Muskie is designed to process multiple views simultaneously and introduce multi-view consistency in pre-training stage. Muskie is trained to reconstruct heavily masked content in one view by finding and utilizing geometric correspondences from other views. Through this pretext task and our proposed aggressive masking strategy, the model implicitly to learn view-invariant features and develop strong geometric understanding without any 3D supervision. Compared with state-of-the-art frame-wise backbones such as DINO, Muskie achieves higher multi-view correspondence accuracy. Furthermore, we demonstrate that using Muskie as a backbone consistently enhances performance on downstream 3D tasks, including camera pose estimation and pointmap reconstruction. Codes are publicly available at https://leo-frank.github.io/Muskie/