Deforming Radiance Fields with Cages

TL;DR

This paper introduces a novel method for deforming radiance fields using a cage-based approach, allowing explicit scene manipulation and animation by manipulating a mesh enclosure around objects.

Contribution

It extends cage-based mesh deformation techniques to radiance fields, enabling free-form scene deformation and rendering of both synthetic and real-world datasets.

Findings

Effective deformation of radiance fields demonstrated on synthetic datasets

Successful application to real-world datasets

Enables scene manipulation and animation in radiance field rendering

Abstract

Recent advances in radiance fields enable photorealistic rendering of static or dynamic 3D scenes, but still do not support explicit deformation that is used for scene manipulation or animation. In this paper, we propose a method that enables a new type of deformation of the radiance field: free-form radiance field deformation. We use a triangular mesh that encloses the foreground object called cage as an interface, and by manipulating the cage vertices, our approach enables the free-form deformation of the radiance field. The core of our approach is cage-based deformation which is commonly used in mesh deformation. We propose a novel formulation to extend it to the radiance field, which maps the position and the view direction of the sampling points from the deformed space to the canonical space, thus enabling the rendering of the deformed scene. The deformation results of the synthetic datasets and the real-world datasets demonstrate the effectiveness of our approach.