Toon3D: Seeing Cartoons from New Perspectives

TL;DR

Toon3D introduces a method to reconstruct 3D structures from cartoon and anime images by deforming images to achieve geometric consistency, enabling novel-view synthesis despite the lack of explicit 3D cues.

Contribution

The paper presents Toon3D, a novel approach that deforms cartoon images to recover consistent 3D structure, addressing the limitations of traditional SfM methods on creative media.

Findings

Outperforms classical SfM in geometric consistency.

Enables reliable camera pose estimation from inconsistent images.

Provides a dataset with annotated multi-view cartoon imagery.

Abstract

We recover the underlying 3D structure from images of cartoons and anime depicting the same scene. This is an interesting problem domain because images in creative media are often depicted without explicit geometric consistency for storytelling and creative expression-they are only 3D in a qualitative sense. While humans can easily perceive the underlying 3D scene from these images, existing Structure-from-Motion (SfM) methods that assume 3D consistency fail catastrophically. We present Toon3D for reconstructing geometrically inconsistent images. Our key insight is to deform the input images while recovering camera poses and scene geometry, effectively explaining away geometrical inconsistencies to achieve consistency. This process is guided by the structure inferred from monocular depth predictions. We curate a dataset with multi-view imagery from cartoons and anime that we annotate with reliable sparse correspondences using our user-friendly annotation tool. Our recovered point clouds can be plugged into novel-view synthesis methods to experience cartoons from viewpoints never drawn before. We evaluate against classical and recent learning-based SfM methods, where Toon3D is able to obtain more reliable camera poses and scene geometry.