Deformable Neural Radiance Fields using RGB and Event Cameras

TL;DR

This paper introduces a novel approach to model deformable neural radiance fields using asynchronous event camera data combined with RGB frames, enabling better modeling of fast-moving, deformable objects in dynamic scenes.

Contribution

It develops a joint optimization method that estimates camera poses and radiance fields from asynchronous event streams and sparse RGB data, addressing high deformation and low acquisition rate challenges.

Findings

Significant improvement over state-of-the-art methods.

Effective modeling of dynamic deformable scenes.

Validated on both synthetic and real-world datasets.

Abstract

Modeling Neural Radiance Fields for fast-moving deformable objects from visual data alone is a challenging problem. A major issue arises due to the high deformation and low acquisition rates. To address this problem, we propose to use event cameras that offer very fast acquisition of visual change in an asynchronous manner. In this work, we develop a novel method to model the deformable neural radiance fields using RGB and event cameras. The proposed method uses the asynchronous stream of events and calibrated sparse RGB frames. In our setup, the camera pose at the individual events required to integrate them into the radiance fields remains unknown. Our method jointly optimizes these poses and the radiance field. This happens efficiently by leveraging the collection of events at once and actively sampling the events during learning. Experiments conducted on both realistically rendered graphics and real-world datasets demonstrate a significant benefit of the proposed method over the state-of-the-art and the compared baseline. This shows a promising direction for modeling deformable neural radiance fields in real-world dynamic scenes.