GGRt: Towards Pose-free Generalizable 3D Gaussian Splatting in Real-time

TL;DR

GGRt introduces a pose-free, real-time 3D Gaussian Splatting framework that estimates camera poses from images, enabling fast, high-resolution view synthesis without real camera pose data.

Contribution

It is the first pose-free generalizable 3D Gaussian Splatting framework that combines joint pose estimation with high-resolution, real-time rendering capabilities.

Findings

Achieves inference at ≥ 5 FPS and real-time rendering at ≥ 100 FPS.

Outperforms existing pose-free NeRF-based methods in speed and effectiveness.

Approaches the performance of pose-based 3D-GS methods on benchmark datasets.

Abstract

This paper presents GGRt, a novel approach to generalizable novel view synthesis that alleviates the need for real camera poses, complexity in processing high-resolution images, and lengthy optimization processes, thus facilitating stronger applicability of 3D Gaussian Splatting (3D-GS) in real-world scenarios. Specifically, we design a novel joint learning framework that consists of an Iterative Pose Optimization Network (IPO-Net) and a Generalizable 3D-Gaussians (G-3DG) model. With the joint learning mechanism, the proposed framework can inherently estimate robust relative pose information from the image observations and thus primarily alleviate the requirement of real camera poses. Moreover, we implement a deferred back-propagation mechanism that enables high-resolution training and inference, overcoming the resolution constraints of previous methods. To enhance the speed and efficiency, we further introduce a progressive Gaussian cache module that dynamically adjusts during training and inference. As the first pose-free generalizable 3D-GS framework, GGRt achieves inference at $\ge$ 5 FPS and real-time rendering at $\ge$ 100 FPS. Through extensive experimentation, we demonstrate that our method outperforms existing NeRF-based pose-free techniques in terms of inference speed and effectiveness. It can also approach the real pose-based 3D-GS methods. Our contributions provide a significant leap forward for the integration of computer vision and computer graphics into practical applications, offering state-of-the-art results on LLFF, KITTI, and Waymo Open datasets and enabling real-time rendering for immersive experiences.