Sat-NeRF: Learning Multi-View Satellite Photogrammetry With Transient Objects and Shadow Modeling Using RPC Cameras

TL;DR

Sat-NeRF is a novel neural rendering model tailored for satellite imagery that effectively handles shadows and transient objects, enabling high-quality multi-view 3D reconstruction from satellite images.

Contribution

It introduces a satellite-specific neural radiance field model that incorporates RPC camera models and robustly manages appearance changes due to shadows and transient objects.

Findings

Achieves surface reconstruction quality comparable to stereo pipelines.

Demonstrates robustness to shadows and transient objects in satellite images.

Improves performance with bundle adjustment of satellite camera models.

Abstract

We introduce the Satellite Neural Radiance Field (Sat-NeRF), a new end-to-end model for learning multi-view satellite photogrammetry in the wild. Sat-NeRF combines some of the latest trends in neural rendering with native satellite camera models, represented by rational polynomial coefficient (RPC) functions. The proposed method renders new views and infers surface models of similar quality to those obtained with traditional state-of-the-art stereo pipelines. Multi-date images exhibit significant changes in appearance, mainly due to varying shadows and transient objects (cars, vegetation). Robustness to these challenges is achieved by a shadow-aware irradiance model and uncertainty weighting to deal with transient phenomena that cannot be explained by the position of the sun. We evaluate Sat-NeRF using WorldView-3 images from different locations and stress the advantages of applying a bundle adjustment to the satellite camera models prior to training. This boosts the network performance and can optionally be used to extract additional cues for depth supervision.