Incorporating Season and Solar Specificity into Renderings made by a NeRF Architecture using Satellite Images

TL;DR

This paper extends NeRF-based satellite image rendering to include seasonality, enabling independent rendering of seasonal features and shadows, with improved accuracy demonstrated on multiple geographic areas.

Contribution

It introduces a method to render season-specific features in NeRFs from satellite images by adding a seasonal input and specialized loss terms, addressing shadow ambiguity issues.

Findings

Accurately renders seasonal features independently of shadows.

Generates reliable height maps and shadow predictions.

Demonstrates effectiveness on eight satellite image areas.

Abstract

As a result of Shadow NeRF and Sat-NeRF, it is possible to take the solar angle into account in a NeRF-based framework for rendering a scene from a novel viewpoint using satellite images for training. Our work extends those contributions and shows how one can make the renderings season-specific. Our main challenge was creating a Neural Radiance Field (NeRF) that could render seasonal features independently of viewing angle and solar angle while still being able to render shadows. We teach our network to render seasonal features by introducing one more input variable -- time of the year. However, the small training datasets typical of satellite imagery can introduce ambiguities in cases where shadows are present in the same location for every image of a particular season. We add additional terms to the loss function to discourage the network from using seasonal features for accounting for shadows. We show the performance of our network on eight Areas of Interest containing images captured by the Maxar WorldView-3 satellite. This evaluation includes tests measuring the ability of our framework to accurately render novel views, generate height maps, predict shadows, and specify seasonal features independently from shadows. Our ablation studies justify the choices made for network design parameters.