USB-NeRF: Unrolling Shutter Bundle Adjusted Neural Radiance Fields
Moyang Li, Peng Wang, Lingzhe Zhao, Bangyan Liao, Peidong Liu

TL;DR
USB-NeRF introduces a novel method to correct rolling shutter distortions in neural radiance fields, enabling accurate 3D scene reconstruction, view synthesis, and camera motion estimation from RS images.
Contribution
It models the physical process of RS cameras within NeRF, allowing simultaneous correction of distortions and accurate camera trajectory recovery.
Findings
Outperforms prior methods in RS distortion removal
Improves quality of novel view synthesis from RS images
Enables high-fidelity global shutter video reconstruction
Abstract
Neural Radiance Fields (NeRF) has received much attention recently due to its impressive capability to represent 3D scene and synthesize novel view images. Existing works usually assume that the input images are captured by a global shutter camera. Thus, rolling shutter (RS) images cannot be trivially applied to an off-the-shelf NeRF algorithm for novel view synthesis. Rolling shutter effect would also affect the accuracy of the camera pose estimation (e.g. via COLMAP), which further prevents the success of NeRF algorithm with RS images. In this paper, we propose Unrolling Shutter Bundle Adjusted Neural Radiance Fields (USB-NeRF). USB-NeRF is able to correct rolling shutter distortions and recover accurate camera motion trajectory simultaneously under the framework of NeRF, by modeling the physical image formation process of a RS camera. Experimental results demonstrate that USB-NeRF…
Peer Reviews
Decision·ICLR 2024 poster
1. The paper is well written and easy to follow. 2. The proposed idea is novel and technically solid. 3. The evaluation is convincing and supports the main contribution well.
I don’t see any major weakness. It would be interesting to see more analysis/discussions on how much is the performance gap when modelling rolling shutter effect vs not modelling with datasets from modern DSLR cameras and smart phone cameras.
The proposed method is evaluated on both synthetic and real world dataset and demonstrated the improvement of reconstruction with and without the rolling shutter correction.
I am not fully convinced that using rolling shutter camera is an effective way to capture a NeRF model. There is actually no motivation/benefits to use rolling shutter camera to capture a NeRF model. Considering the case that using rolling shutter camera is necessary, the proposed solution is just a simple two-step approach with first rolling shutter correction followed by NeRF reconstruction. I do not see any connection between rolling shutter correction and NeRF reconstruction in the proposed
1. This paper is very well written. I can easily understand the paper even though I'm not very familiar with the rolling-shutter camera. 2. The proposed method is simple yet effective, which uses the cubic B-Spline to interpolate between camera poses instead of linear interpolation. 3. The paper did exhaustive experiments to evaluate the effectiveness of their method on both the synthetic and real-world datasets. Though there is a lack of baseline methods for rolling-shutter NeRF, they compare
1. I think the ATE (absolute trajectory error) in Table 3 is the same as the absolute translation error (I'm used to the term `translation` instead of `trajectory`). Therefore, only the translation errors are given and no rotation errors are provided. Moreover, the unit of the ATE is unclear (I think it is in meters). 2. The cubic B-Splines interpolation is suitable for complex camera trajectories, however, it can be worse than the linear interpolation method when the camera moves at a constant
Code & Models
Videos
Taxonomy
TopicsAdvanced Vision and Imaging · Optical measurement and interference techniques · Image Processing Techniques and Applications
