Variational Bayes Gaussian Splatting
Toon Van de Maele, Ozan Catal, Alexander Tschantz, Christopher L. Buckley, Tim Verbelen
TL;DR
This paper introduces Variational Bayes Gaussian Splatting (VBGS), a novel method for 3D scene modeling that enables efficient continual learning from streaming data by framing the process as variational inference, avoiding catastrophic forgetting.
Contribution
The paper proposes a variational inference framework for Gaussian Splatting that allows efficient, sequential updates without replay buffers, improving continual learning capabilities.
Findings
VBGS matches state-of-the-art on static datasets.
VBGS enables effective continual learning from streamed data.
The method avoids catastrophic forgetting in 3D scene modeling.
Abstract
Recently, 3D Gaussian Splatting has emerged as a promising approach for modeling 3D scenes using mixtures of Gaussians. The predominant optimization method for these models relies on backpropagating gradients through a differentiable rendering pipeline, which struggles with catastrophic forgetting when dealing with continuous streams of data. To address this limitation, we propose Variational Bayes Gaussian Splatting (VBGS), a novel approach that frames training a Gaussian splat as variational inference over model parameters. By leveraging the conjugacy properties of multivariate Gaussians, we derive a closed-form variational update rule, allowing efficient updates from partial, sequential observations without the need for replay buffers. Our experiments show that VBGS not only matches state-of-the-art performance on static datasets, but also enables continual learning from sequentially…
Peer Reviews
Decision·Submitted to ICLR 2025
**S1.** The method addresses an important limitation of 3DGS. **S2.** The method is principled and presented with clarity. **S3.** The experimental validation lends support to the practical usefulness of the method: in particular, in the online setting 3DGS is clearly impacted by catastrophic forgetting while VBGS is not.
**W1.** The method is motivated by the application to continual learning scenarios such as SLAM. However, the experimental validation does not compare against recent SLAM methods or datasets, e.g., (Matsuki et al., 2024; Keetha et al., 2024). **W2.** The significance of the mean-field approximation is not discussed or investigated experimentally. **W3.** The method's reliance on depth maps is, as acknowledged by the authors, a limitation of the method. The author’s propose inferring depth maps
The main strengths I find in this paper lie in its novelty, clarity in writing, and reproducibility. - Both relating 3D Gaussian splatting as Gaussian mixture model and variational inference is new to this field, and this novelty should be acknowledged. - Application of continual learning well justifies the usage of variational inference framework. - The training scheme is described in detail and the described motivation and methodology is quite clear. - The supplementary material has the code
Although I find the method informative, there are several points to be clarified in order to improve the presentation. - Common 3DGS involves several millions of Gaussians, but the presented experiments are only based on 100k Gaussians, which is very slim. This small parameter constraint makes the PSNR scores in Table 1 way below the numbers reported by other NeRF/3DGS papers which go high above 30 dB for the same NeRF synthetic dataset. Is there any specific reason (e.g., computational burden)
1. Effectively handles continuous data streams: By framing the training process as variational inference over model parameters, VBGS avoids the issue of catastrophic forgetting typically seen in continuous data streams with traditional methods. 2. No need for replay buffers: VBGS leverages the conjugacy properties of multivariate Gaussians, deriving a closed-form variational update rule that allows efficient incremental updates without requiring replay buffers.
1. Limited improvement. In Tab. 1, your method is comparable to the Gradient method (3DGS), which cannot prove the effectiveness of your method. 2. Unlike 3D Gaussian Splatting (3DGS), it relies on RGBD data. It means your method needs a depth camera and does not outperform 3DGS. 3. The experiments are not comprehensive, as they do not include evaluations on standard datasets like mip-NeRF 360, which could provide a more robust comparison with existing methods. 4. The comparison with baseline
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsFault Detection and Control Systems
MethodsVariational Inference