DreamCreature: Crafting Photorealistic Virtual Creatures from Imagination

TL;DR

DreamCreature introduces an unsupervised method to generate photorealistic, novel virtual creatures by composing learned sub-concepts, advancing digital asset creation and biodiversity analysis.

Contribution

The paper presents DreamCreature, a novel approach that unsupervisedly extracts sub-concepts to generate new, detailed virtual creatures with high fidelity and flexibility.

Findings

Outperforms prior methods in qualitative evaluations

Achieves higher quantitative scores on image benchmarks

Enables diverse creative applications

Abstract

Recent text-to-image (T2I) generative models allow for high-quality synthesis following either text instructions or visual examples. Despite their capabilities, these models face limitations in creating new, detailed creatures within specific categories (e.g., virtual dog or bird species), which are valuable in digital asset creation and biodiversity analysis. To bridge this gap, we introduce a novel task, Virtual Creatures Generation: Given a set of unlabeled images of the target concepts (e.g., 200 bird species), we aim to train a T2I model capable of creating new, hybrid concepts within diverse backgrounds and contexts. We propose a new method called DreamCreature, which identifies and extracts the underlying sub-concepts (e.g., body parts of a specific species) in an unsupervised manner. The T2I thus adapts to generate novel concepts (e.g., new bird species) with faithful structures and photorealistic appearance by seamlessly and flexibly composing learned sub-concepts. To enhance sub-concept fidelity and disentanglement, we extend the textual inversion technique by incorporating an additional projector and tailored attention loss regularization. Extensive experiments on two fine-grained image benchmarks demonstrate the superiority of DreamCreature over prior methods in both qualitative and quantitative evaluation. Ultimately, the learned sub-concepts facilitate diverse creative applications, including innovative consumer product designs and nuanced property modifications.