Sketch-guided Cage-based 3D Gaussian Splatting Deformation

TL;DR

This paper introduces a sketch-guided deformation system for 3D Gaussian Splatting models, enabling intuitive, fine-grained, and semantically plausible modifications from a single viewpoint, enhancing 3D editing capabilities.

Contribution

It presents a novel deformation method combining cage-based techniques with Neural Jacobian Fields, integrating diffusion priors for semantic plausibility, and demonstrating effective model animation.

Findings

Effective fine-grained deformation control demonstrated

Semantic plausibility achieved via diffusion priors

Application in animating static 3D models

Abstract

3D Gaussian Splatting (GS) is one of the most promising novel 3D representations that has received great interest in computer graphics and computer vision. While various systems have introduced editing capabilities for 3D GS, such as those guided by text prompts, fine-grained control over deformation remains an open challenge. In this work, we present a novel sketch-guided 3D GS deformation system that allows users to intuitively modify the geometry of a 3D GS model by drawing a silhouette sketch from a single viewpoint. Our approach introduces a new deformation method that combines cage-based deformations with a variant of Neural Jacobian Fields, enabling precise, fine-grained control. Additionally, it leverages large-scale 2D diffusion priors and ControlNet to ensure the generated deformations are semantically plausible. Through a series of experiments, we demonstrate the effectiveness of our method and showcase its ability to animate static 3D GS models as one of its key applications.