Factorized Diffusion: Perceptual Illusions by Noise Decomposition

TL;DR

This paper introduces a zero-shot diffusion-based method for image decomposition into multiple components, enabling control over image appearance based on viewing conditions, with applications to hybrid image generation and real image editing.

Contribution

It presents a novel noise decomposition technique in diffusion models for multi-component image control and extends to real image hybridization and inverse problems.

Findings

Effective decomposition into frequency, grayscale, and motion blur components.

Ability to generate hybrid images with multiple prompts.

Extension to real image hybridization and inverse problem solving.

Abstract

Given a factorization of an image into a sum of linear components, we present a zero-shot method to control each individual component through diffusion model sampling. For example, we can decompose an image into low and high spatial frequencies and condition these components on different text prompts. This produces hybrid images, which change appearance depending on viewing distance. By decomposing an image into three frequency subbands, we can generate hybrid images with three prompts. We also use a decomposition into grayscale and color components to produce images whose appearance changes when they are viewed in grayscale, a phenomena that naturally occurs under dim lighting. And we explore a decomposition by a motion blur kernel, which produces images that change appearance under motion blurring. Our method works by denoising with a composite noise estimate, built from the components of noise estimates conditioned on different prompts. We also show that for certain decompositions, our method recovers prior approaches to compositional generation and spatial control. Finally, we show that we can extend our approach to generate hybrid images from real images. We do this by holding one component fixed and generating the remaining components, effectively solving an inverse problem.