KFD-NeRF: Rethinking Dynamic NeRF with Kalman Filter

TL;DR

KFD-NeRF introduces a Kalman filter-based dynamic neural radiance field that models scene motion as a dynamic system, enabling high-quality, efficient view synthesis for both synthetic and real data.

Contribution

It presents a novel Kalman filter-guided deformation framework integrated with a tri-plane representation for dynamic NeRFs, improving accuracy and efficiency.

Findings

Achieves state-of-the-art view synthesis performance.

Demonstrates high-quality results with faster convergence.

Performs comparably or better than previous methods in experiments.

Abstract

We introduce KFD-NeRF, a novel dynamic neural radiance field integrated with an efficient and high-quality motion reconstruction framework based on Kalman filtering. Our key idea is to model the dynamic radiance field as a dynamic system whose temporally varying states are estimated based on two sources of knowledge: observations and predictions. We introduce a novel plug-in Kalman filter guided deformation field that enables accurate deformation estimation from scene observations and predictions. We use a shallow Multi-Layer Perceptron (MLP) for observations and model the motion as locally linear to calculate predictions with motion equations. To further enhance the performance of the observation MLP, we introduce regularization in the canonical space to facilitate the network's ability to learn warping for different frames. Additionally, we employ an efficient tri-plane representation for encoding the canonical space, which has been experimentally demonstrated to converge quickly with high quality. This enables us to use a shallower observation MLP, consisting of just two layers in our implementation. We conduct experiments on synthetic and real data and compare with past dynamic NeRF methods. Our KFD-NeRF demonstrates similar or even superior rendering performance within comparable computational time and achieves state-of-the-art view synthesis performance with thorough training.