Varying Manifolds in Diffusion: From Time-varying Geometries to Visual Saliency

TL;DR

This paper explores the geometric properties of diffusion models, introducing the concept of generation rate to relate manifold deformation to visual saliency, enabling advanced image manipulation techniques.

Contribution

It introduces the generation rate as a measure of manifold deformation in diffusion models and develops a differentiable scheme for image manipulation based on generation curves.

Findings

Generation rate correlates with visual saliency.

The proposed method improves image manipulation results.

Framework is effective across various manipulation tasks.

Abstract

Deep generative models learn the data distribution, which is concentrated on a low-dimensional manifold. The geometric analysis of distribution transformation provides a better understanding of data structure and enables a variety of applications. In this paper, we study the geometric properties of the diffusion model, whose forward diffusion process and reverse generation process construct a series of distributions on manifolds which vary over time. Our key contribution is the introduction of generation rate, which corresponds to the local deformation of manifold over time around an image component. We show that the generation rate is highly correlated with intuitive visual properties, such as visual saliency, of the image component. Further, we propose an efficient and differentiable scheme to estimate the generation rate for a given image component over time, giving rise to a generation curve. The differentiable nature of our scheme allows us to control the shape of the generation curve via optimization. Using different loss functions, our generation curve matching algorithm provides a unified framework for a range of image manipulation tasks, including semantic transfer, object removal, saliency manipulation, image blending, etc. We conduct comprehensive analytical evaluations to support our findings and evaluate our framework on various manipulation tasks. The results show that our method consistently leads to better manipulation results, compared to recent baselines.