MoE3D: A Mixture-of-Experts Module for 3D Reconstruction

TL;DR

This paper introduces MoE3D, a mixture-of-experts module that improves 3D reconstruction accuracy by reducing boundary artifacts and handling depth discontinuities more effectively, with minimal additional computational cost.

Contribution

We propose a mixture-of-experts formulation integrated into existing 3D models to better handle depth boundaries and improve reconstruction quality.

Findings

Significant reduction in boundary artifacts.

Improved overall reconstruction accuracy.

Effective even with limited fine-tuning data.

Abstract

We propose a simple yet effective approach to enhance the performance of feed-forward 3D reconstruction models. Existing methods often struggle near depth discontinuities, where standard regression losses encourage spatial averaging and thus blur sharp boundaries. To address this issue, we introduce a mixture-of-experts formulation that handles uncertainty at depth boundaries by combining multiple smooth depth predictions. A softmax weighting head dynamically selects among these hypotheses on a per-pixel basis. By integrating our mixture model into a pre-trained state-of-the-art 3D model, we achieve a substantial reduction of boundary artifacts and gains in overall reconstruction accuracy. Notably, our approach is highly compute efficient, delivering generalizable improvements even when fine-tuned on a small subset of training data while incurring only negligible additional inference computation, suggesting a promising direction for lightweight and accurate 3D reconstruction.