Illusion3D: 3D Multiview Illusion with 2D Diffusion Priors

TL;DR

This paper introduces Illusion3D, a novel method that creates 3D multiview illusions from 2D diffusion priors, enabling intricate and versatile visual illusions with multiple perspectives.

Contribution

It presents a new approach that leverages pre-trained diffusion models and differentiable rendering to generate complex 3D multiview illusions from text or images.

Findings

Effective generation of diverse 3D illusions

Improved quality through novel techniques

Demonstrated versatility with various 3D forms

Abstract

Automatically generating multiview illusions is a compelling challenge, where a single piece of visual content offers distinct interpretations from different viewing perspectives. Traditional methods, such as shadow art and wire art, create interesting 3D illusions but are limited to simple visual outputs (i.e., figure-ground or line drawing), restricting their artistic expressiveness and practical versatility. Recent diffusion-based illusion generation methods can generate more intricate designs but are confined to 2D images. In this work, we present a simple yet effective approach for creating 3D multiview illusions based on user-provided text prompts or images. Our method leverages a pre-trained text-to-image diffusion model to optimize the textures and geometry of neural 3D representations through differentiable rendering. When viewed from multiple angles, this produces different interpretations. We develop several techniques to improve the quality of the generated 3D multiview illusions. We demonstrate the effectiveness of our approach through extensive experiments and showcase illusion generation with diverse 3D forms.