Divide and Compose with Score Based Generative Models

TL;DR

This paper introduces a method to learn and manipulate image components in an unsupervised manner using score-based generative models, enabling more controlled and interpretable image editing and composition.

Contribution

It proposes a novel decomposition of score functions into components representing image parts, inspired by energy-based models, for improved image manipulation.

Findings

Score components can be visualized and interpreted as image parts.

The method enables composition and editing of images by manipulating learned components.

Unsupervised learning of image components improves controllability in generative models.

Abstract

While score based generative models, or diffusion models, have found success in image synthesis, they are often coupled with text data or image label to be able to manipulate and conditionally generate images. Even though manipulation of images by changing the text prompt is possible, our understanding of the text embedding and our ability to modify it to edit images is quite limited. Towards the direction of having more control over image manipulation and conditional generation, we propose to learn image components in an unsupervised manner so that we can compose those components to generate and manipulate images in informed manner. Taking inspiration from energy based models, we interpret different score components as the gradient of different energy functions. We show how score based learning allows us to learn interesting components and we can visualize them through generation. We also show how this novel decomposition allows us to compose, generate and modify images in interesting ways akin to dreaming. We make our code available at https://github.com/sandeshgh/Score-based-disentanglement