DeepSurfels: Learning Online Appearance Fusion

TL;DR

DeepSurfels introduces a hybrid scene representation that effectively encodes geometry and high-frequency textures, enabling online appearance updates and superior integration with machine learning, outperforming classical and recent methods.

Contribution

The paper proposes DeepSurfels, a novel hybrid scene representation combining explicit and neural components for improved online appearance fusion.

Findings

Better representation of high-frequency textures.

Lower runtime and improved scalability.

Enhanced generalization over existing methods.

Abstract

We present DeepSurfels, a novel hybrid scene representation for geometry and appearance information. DeepSurfels combines explicit and neural building blocks to jointly encode geometry and appearance information. In contrast to established representations, DeepSurfels better represents high-frequency textures, is well-suited for online updates of appearance information, and can be easily combined with machine learning methods. We further present an end-to-end trainable online appearance fusion pipeline that fuses information from RGB images into the proposed scene representation and is trained using self-supervision imposed by the reprojection error with respect to the input images. Our method compares favorably to classical texture mapping approaches as well as recent learning-based techniques. Moreover, we demonstrate lower runtime, im-proved generalization capabilities, and better scalability to larger scenes compared to existing methods.