Self-supervised novel 2D view synthesis of large-scale scenes with efficient multi-scale voxel carving

TL;DR

This paper introduces a self-supervised, multi-scale voxel carving approach for realistic 2D view synthesis of large-scale, real-world scenes, effectively handling noise and variations in pose, depth, and illumination.

Contribution

It presents a novel multi-scale voxel carving method for real-world large-scale scenes and a self-training framework for high-resolution view synthesis.

Findings

Outperforms current state-of-the-art on complex large-scale scenes

Effectively handles noise in pose, depth, and illumination

Demonstrates applicability to real UAV environments

Abstract

The task of generating novel views of real scenes is increasingly important nowadays when AI models become able to create realistic new worlds. In many practical applications, it is important for novel view synthesis methods to stay grounded in the physical world as much as possible, while also being able to imagine it from previously unseen views. While most current methods are developed and tested in virtual environments with small scenes and no errors in pose and depth information, we push the boundaries to the real-world domain of large scales in the new context of UAVs. Our algorithmic contributions are two folds. First, we manage to stay anchored in the real 3D world, by introducing an efficient multi-scale voxel carving method, which is able to accommodate significant noises in pose, depth, and illumination variations, while being able to reconstruct the view of the world from drastically different poses at test time. Second, our final high-resolution output is efficiently self-trained on data automatically generated by the voxel carving module, which gives it the flexibility to adapt efficiently to any scene. We demonstrated the effectiveness of our method on highly complex and large-scale scenes in real environments while outperforming the current state-of-the-art. Our code is publicly available: https://github.com/onorabil/MSVC.