AutoPoly: Predicting a Polygonal Mesh Construction Sequence from a Silhouette Image

TL;DR

AutoPoly is a hybrid approach that predicts a sequence of polygonal mesh construction steps from a silhouette image, enabling automatic and expedited 3D modeling by combining MCTS with deep learning components.

Contribution

It introduces a novel hybrid method using MCTS and differentiable rendering to predict topological and geometric actions for polygonal shape modeling from silhouettes.

Findings

Effective on multiple man-made object categories

Can alter topology unlike previous fixed-topology methods

Improves modeling efficiency through learned action prediction

Abstract

Polygonal modeling is a core task of content creation in Computer Graphics. The complexity of modeling, in terms of the number and the order of operations and time required to execute them makes it challenging to learn and execute. Our goal is to automatically derive a polygonal modeling sequence for a given target. Then, one can learn polygonal modeling by observing the resulting sequence and also expedite the modeling process by starting from the auto-generated result. As a starting point for building a system for 3D modeling in the future, we tackle the 2D shape modeling problem and present AutoPoly, a hybrid method that generates a polygonal mesh construction sequence from a silhouette image. The key idea of our method is the use of the Monte Carlo tree search (MCTS) algorithm and differentiable rendering to separately predict sequential topological actions and geometric actions. Our hybrid method can alter topology, whereas the recently proposed inverse shape estimation methods using differentiable rendering can only handle a fixed topology. Our novel reward function encourages MCTS to select topological actions that lead to a simpler shape without self-intersection. We further designed two deep learning-based methods to improve the expansion and simulation steps in the MCTS search process: an $n$-step "future action prediction" network (nFAP-Net) to generate candidates for potential topological actions, and a shape warping network (WarpNet) to predict polygonal shapes given the predicted rendered images and topological actions. We demonstrate the efficiency of our method on 2D polygonal shapes of multiple man-made object categories.