Deformable NeRF using Recursively Subdivided Tetrahedra

TL;DR

DeformRF introduces a two-stage training approach with recursively subdivided tetrahedra to enable explicit object deformation in neural radiance fields, balancing mesh quality and computational efficiency for high-quality view synthesis.

Contribution

The paper presents a novel two-stage training strategy with recursive tetrahedral subdivision to improve deformable NeRF representations, reducing computational costs and handling complex geometries effectively.

Findings

Effective deformation and view synthesis demonstrated on synthetic datasets.

High-quality rendering achieved with multi-resolution tetrahedral encoding.

Reduced tetrahedralization complexity compared to existing methods.

Abstract

While neural radiance fields (NeRF) have shown promise in novel view synthesis, their implicit representation limits explicit control over object manipulation. Existing research has proposed the integration of explicit geometric proxies to enable deformation. However, these methods face two primary challenges: firstly, the time-consuming and computationally demanding tetrahedralization process; and secondly, handling complex or thin structures often leads to either excessive, storage-intensive tetrahedral meshes or poor-quality ones that impair deformation capabilities. To address these challenges, we propose DeformRF, a method that seamlessly integrates the manipulability of tetrahedral meshes with the high-quality rendering capabilities of feature grid representations. To avoid ill-shaped tetrahedra and tetrahedralization for each object, we propose a two-stage training strategy. Starting with an almost-regular tetrahedral grid, our model initially retains key tetrahedra surrounding the object and subsequently refines object details using finer-granularity mesh in the second stage. We also present the concept of recursively subdivided tetrahedra to create higher-resolution meshes implicitly. This enables multi-resolution encoding while only necessitating the storage of the coarse tetrahedral mesh generated in the first training stage. We conduct a comprehensive evaluation of our DeformRF on both synthetic and real-captured datasets. Both quantitative and qualitative results demonstrate the effectiveness of our method for novel view synthesis and deformation tasks. Project page: https://ustc3dv.github.io/DeformRF/