Radiance Field Learners As UAV First-Person Viewers

TL;DR

FPV-NeRF is a novel neural radiance field approach designed for UAV first-person view videos, addressing challenges of limited viewpoints and scale variations by ensuring temporal consistency, global structure preservation, and multi-scale scene representation, and includes a new dataset and view synthesis method.

Contribution

The paper introduces FPV-NeRF, a comprehensive framework for UAV FPV scene reconstruction that incorporates temporal, global, and local features, along with a new dataset and view synthesis technique.

Findings

Outperforms state-of-the-art methods on UAV datasets

Effectively captures multi-scale scene details

Enhances spatial perception for UAV navigation

Abstract

First-Person-View (FPV) holds immense potential for revolutionizing the trajectory of Unmanned Aerial Vehicles (UAVs), offering an exhilarating avenue for navigating complex building structures. Yet, traditional Neural Radiance Field (NeRF) methods face challenges such as sampling single points per iteration and requiring an extensive array of views for supervision. UAV videos exacerbate these issues with limited viewpoints and significant spatial scale variations, resulting in inadequate detail rendering across diverse scales. In response, we introduce FPV-NeRF, addressing these challenges through three key facets: (1) Temporal consistency. Leveraging spatio-temporal continuity ensures seamless coherence between frames; (2) Global structure. Incorporating various global features during point sampling preserves space integrity; (3) Local granularity. Employing a comprehensive framework and multi-resolution supervision for multi-scale scene feature representation tackles the intricacies of UAV video spatial scales. Additionally, due to the scarcity of publicly available FPV videos, we introduce an innovative view synthesis method using NeRF to generate FPV perspectives from UAV footage, enhancing spatial perception for drones. Our novel dataset spans diverse trajectories, from outdoor to indoor environments, in the UAV domain, differing significantly from traditional NeRF scenarios. Through extensive experiments encompassing both interior and exterior building structures, FPV-NeRF demonstrates a superior understanding of the UAV flying space, outperforming state-of-the-art methods in our curated UAV dataset. Explore our project page for further insights: https://fpv-nerf.github.io/.