Generating Synthetic Stereo Datasets using 3D Gaussian Splatting and Expert Knowledge Transfer

TL;DR

This paper presents a novel pipeline using 3D Gaussian Splatting and expert knowledge transfer for efficient synthetic stereo dataset generation, enabling fast fine-tuning and competitive zero-shot performance.

Contribution

The paper introduces a 3D Gaussian Splatting-based method for stereo dataset creation that outperforms NeRF-based approaches in efficiency and offers insights into geometry reconstruction and robustness.

Findings

3DGS-based datasets enable competitive zero-shot stereo model performance.

Reconstructed geometry from 3DGS can be noisy, affecting model training.

FoundationStereo disparity estimates provide cleaner geometry for better results.

Abstract

In this paper, we introduce a 3D Gaussian Splatting (3DGS)-based pipeline for stereo dataset generation, offering an efficient alternative to Neural Radiance Fields (NeRF)-based methods. To obtain useful geometry estimates, we explore utilizing the reconstructed geometry from the explicit 3D representations as well as depth estimates from the FoundationStereo model in an expert knowledge transfer setup. We find that when fine-tuning stereo models on 3DGS-generated datasets, we demonstrate competitive performance in zero-shot generalization benchmarks. When using the reconstructed geometry directly, we observe that it is often noisy and contains artifacts, which propagate noise to the trained model. In contrast, we find that the disparity estimates from FoundationStereo are cleaner and consequently result in a better performance on the zero-shot generalization benchmarks. Our method highlights the potential for low-cost, high-fidelity dataset creation and fast fine-tuning for deep stereo models. Moreover, we also reveal that while the latest Gaussian Splatting based methods have achieved superior performance on established benchmarks, their robustness falls short in challenging in-the-wild settings warranting further exploration.