RS-NeRF: Neural Radiance Fields from Rolling Shutter Images

TL;DR

RS-NeRF introduces a novel approach to synthesize images from novel views by modeling and correcting rolling shutter distortions in neural radiance fields, significantly improving view synthesis quality in RS-affected images.

Contribution

The paper proposes RS-NeRF, a new method that jointly optimizes NeRF parameters and camera extrinsics to correct rolling shutter effects during view synthesis.

Findings

RS-NeRF outperforms previous methods in synthetic scenarios.

It effectively corrects RS distortions in real-world images.

The multi-sampling algorithm enhances model performance.

Abstract

Neural Radiance Fields (NeRFs) have become increasingly popular because of their impressive ability for novel view synthesis. However, their effectiveness is hindered by the Rolling Shutter (RS) effects commonly found in most camera systems. To solve this, we present RS-NeRF, a method designed to synthesize normal images from novel views using input with RS distortions. This involves a physical model that replicates the image formation process under RS conditions and jointly optimizes NeRF parameters and camera extrinsic for each image row. We further address the inherent shortcomings of the basic RS-NeRF model by delving into the RS characteristics and developing algorithms to enhance its functionality. First, we impose a smoothness regularization to better estimate trajectories and improve the synthesis quality, in line with the camera movement prior. We also identify and address a fundamental flaw in the vanilla RS model by introducing a multi-sampling algorithm. This new approach improves the model's performance by comprehensively exploiting the RGB data across different rows for each intermediate camera pose. Through rigorous experimentation, we demonstrate that RS-NeRF surpasses previous methods in both synthetic and real-world scenarios, proving its ability to correct RS-related distortions effectively. Codes and data available: https://github.com/MyNiuuu/RS-NeRF