Neural Point Catacaustics for Novel-View Synthesis of Reflections

TL;DR

This paper introduces a neural point-based method for synthesizing novel views of scenes with curved reflectors, effectively modeling complex reflections and enabling interactive editing and manipulation of reflections from casual photos.

Contribution

It presents a novel point-based neural warp field for modeling and rendering curved reflections, with explicit reflection representation and scene manipulation capabilities.

Findings

High-quality novel view synthesis with accurate reflections.

Supports reflection editing and scene manipulation.

Enables interactive rendering from casual photos.

Abstract

View-dependent effects such as reflections pose a substantial challenge for image-based and neural rendering algorithms. Above all, curved reflectors are particularly hard, as they lead to highly non-linear reflection flows as the camera moves. We introduce a new point-based representation to compute Neural Point Catacaustics allowing novel-view synthesis of scenes with curved reflectors, from a set of casually-captured input photos. At the core of our method is a neural warp field that models catacaustic trajectories of reflections, so complex specular effects can be rendered using efficient point splatting in conjunction with a neural renderer. One of our key contributions is the explicit representation of reflections with a reflection point cloud which is displaced by the neural warp field, and a primary point cloud which is optimized to represent the rest of the scene. After a short manual annotation step, our approach allows interactive high-quality renderings of novel views with accurate reflection flow. Additionally, the explicit representation of reflection flow supports several forms of scene manipulation in captured scenes, such as reflection editing, cloning of specular objects, reflection tracking across views, and comfortable stereo viewing. We provide the source code and other supplemental material on https://repo-sam.inria.fr/ fungraph/neural_catacaustics/