SC-OmniGS: Self-Calibrating Omnidirectional Gaussian Splatting
Huajian Huang, Yingshu Chen, Longwei Li, Hui Cheng, Tristan Braud,, Yajie Zhao, Sai-Kit Yeung
TL;DR
SC-OmniGS introduces a self-calibrating system for omnidirectional radiance field reconstruction directly from 360-degree images, effectively handling distortions and pose uncertainties for high-quality 3D scene modeling.
Contribution
It presents a novel mathematical framework and differentiable camera model for direct omnidirectional calibration and Gaussian splatting, improving reconstruction accuracy without prior pose information.
Findings
Successfully reconstructs radiance fields from noisy or unknown poses.
Achieves high-quality 3D reconstructions in challenging wide-baseline scenarios.
Reduces distortion effects in real-world 360-degree images.
Abstract
360-degree cameras streamline data collection for radiance field 3D reconstruction by capturing comprehensive scene data. However, traditional radiance field methods do not address the specific challenges inherent to 360-degree images. We present SC-OmniGS, a novel self-calibrating omnidirectional Gaussian splatting system for fast and accurate omnidirectional radiance field reconstruction using 360-degree images. Rather than converting 360-degree images to cube maps and performing perspective image calibration, we treat 360-degree images as a whole sphere and derive a mathematical framework that enables direct omnidirectional camera pose calibration accompanied by 3D Gaussians optimization. Furthermore, we introduce a differentiable omnidirectional camera model in order to rectify the distortion of real-world data for performance enhancement. Overall, the omnidirectional camera…
Peer Reviews
Decision·ICLR 2025 Poster
### Self calibration + omnidirectional images The proposed method would be the first attempt to combine the self-calibration and omnidirectional GS. ### Backward gradient for spherical images For pose refinement of spherical images, the paper derives the gradients in Eqs. (13--14). This would be a technical novelty in this paper.
### Limited technical improvement Self-calibration of GS has been well-studied so far. Also, omnidirectional GS is existing. The proposed system would be practical, but the scientific motivation for combining those two is not quite large, i.e., the technical novelty is limited. ### Gradient derivation The key technical part of the paper, the derivation of gradients on camera poses of spherical images (Eqs. (13--14)), is rather straightforward. This would be naturally extended from perspective c
- The paper is well-structured and easy to follow, making the methodology and findings accessible to readers. - The introduction of a differentiable omnidirectional camera model that enables ray-wise distortion and the derivation of gradients for pose optimization are valuable innovations that expand the applicability of the system. - The use of spherical weights in the photometric loss ensures spatially balanced optimization, which enhances the robustness and accuracy of the optimization proces
Misleading Terminology in Title: While the title suggests "self-calibrating omnidirectional Gaussian splatting," the approach relies on initialization from a structure-from-motion (SfM) pipeline rather than directly calibrating intrinsic parameters from the images alone. This approach is more accurately an optimization process rather than an auto-calibration technique in the classical sense (e.g., auto-calibration from absolute dual quadrics in multiple-view geometry).
(1) The paper proposes a novel self-calibrating method that extends the omnidirectional Gaussian splatting to handle unposed or noisy 360-degree images. (2) The paper introduces a differentiable omnidirectional camera model, which uses trainable focal length and angle distortion coefficients to represent camera distortion. (3) The proposed method achieves state-of-the-art performance in novel view synthesis.
(1) The method estimates camera poses but lacks comparisons of pose accuracy. I think that comparing only rendering quality, especially with NeRF-based calibration methods, is insufficient to determine whether the superior performance of this paper is due to pose optimization, the camera model, or the scene representation using 3D GS. Therefore, I suggest adding quantitative and qualitative comparisons of camera poses on two datasets. (2) The experiment only compared with NeRF-based calibratio
Videos
Taxonomy
TopicsSpectroscopy Techniques in Biomedical and Chemical Research