Pop-Out Motion: 3D-Aware Image Deformation via Learning the Shape Laplacian

TL;DR

This paper introduces a 3D-aware image deformation framework that predicts shape Laplacians from 3D reconstructions to enable realistic, category-agnostic deformations of 2D images with minimal topological errors.

Contribution

It presents a supervised learning approach to predict shape Laplacians for 3D reconstructions, improving deformation plausibility and accuracy over existing methods.

Findings

More accurate deformation weights than alternative methods

Able to deform diverse categories like characters and clothed humans

Produces realistic deformations with minimal topological errors

Abstract

We propose a framework that can deform an object in a 2D image as it exists in 3D space. Most existing methods for 3D-aware image manipulation are limited to (1) only changing the global scene information or depth, or (2) manipulating an object of specific categories. In this paper, we present a 3D-aware image deformation method with minimal restrictions on shape category and deformation type. While our framework leverages 2D-to-3D reconstruction, we argue that reconstruction is not sufficient for realistic deformations due to the vulnerability to topological errors. Thus, we propose to take a supervised learning-based approach to predict the shape Laplacian of the underlying volume of a 3D reconstruction represented as a point cloud. Given the deformation energy calculated using the predicted shape Laplacian and user-defined deformation handles (e.g., keypoints), we obtain bounded biharmonic weights to model plausible handle-based image deformation. In the experiments, we present our results of deforming 2D character and clothed human images. We also quantitatively show that our approach can produce more accurate deformation weights compared to alternative methods (i.e., mesh reconstruction and point cloud Laplacian methods).