OmniSplat: Taming Feed-Forward 3D Gaussian Splatting for Omnidirectional Images with Editable Capabilities

TL;DR

OmniSplat introduces a fast, training-free framework for generating high-quality, editable omnidirectional images using 3D Gaussian splatting, leveraging a Yin-Yang grid to adapt perspective image techniques.

Contribution

It presents a novel, training-free method that effectively adapts 3D Gaussian splatting for omnidirectional images using a Yin-Yang grid, improving quality and efficiency.

Findings

Higher reconstruction accuracy than perspective-trained models

Effective use of Yin-Yang grid for domain adaptation

Fast, training-free generation of omnidirectional images

Abstract

Feed-forward 3D Gaussian splatting (3DGS) models have gained significant popularity due to their ability to generate scenes immediately without needing per-scene optimization. Although omnidirectional images are becoming more popular since they reduce the computation required for image stitching to composite a holistic scene, existing feed-forward models are only designed for perspective images. The unique optical properties of omnidirectional images make it difficult for feature encoders to correctly understand the context of the image and make the Gaussian non-uniform in space, which hinders the image quality synthesized from novel views. We propose OmniSplat, a training-free fast feed-forward 3DGS generation framework for omnidirectional images. We adopt a Yin-Yang grid and decompose images based on it to reduce the domain gap between omnidirectional and perspective images. The Yin-Yang grid can use the existing CNN structure as it is, but its quasi-uniform characteristic allows the decomposed image to be similar to a perspective image, so it can exploit the strong prior knowledge of the learned feed-forward network. OmniSplat demonstrates higher reconstruction accuracy than existing feed-forward networks trained on perspective images. Our project page is available on: https://robot0321.github.io/omnisplat/index.html.