FilterPrompt: A Simple yet Efficient Approach to Guide Image Appearance Transfer in Diffusion Models

TL;DR

FilterPrompt introduces a universal method to improve controllable image generation in diffusion models by manipulating pixel-space features, enabling more precise and flexible control over generated image attributes.

Contribution

The paper presents FilterPrompt, a novel pixel-space operation that enhances controllability in diffusion models, addressing limitations of feature-space disentanglement methods.

Findings

Optimizes feature correlation in generated images

Reduces content conflicts during diffusion process

Improves control over specific image features

Abstract

In controllable generation tasks, flexibly manipulating the generated images to attain a desired appearance or structure based on a single input image cue remains a critical and longstanding challenge. Achieving this requires the effective decoupling of key attributes within the input image data to achieve representations accurately. Previous works have concentrated predominantly on disentangling image attributes within feature space. However, the complex distribution present in real-world data often makes the application of such decoupling algorithms to other datasets challenging. Moreover, the granularity of control over feature encoding frequently fails to meet specific task requirements. Upon scrutinizing the characteristics of various generative models, we have observed that the input sensitivity and dynamic evolution properties of the diffusion model can be effectively fused with the explicit decomposition operation in pixel space. This allows the operation that we design and use in pixel space to achieve the desired control effect on the specific representation in the generated results. Therefore, we propose FilterPrompt, an approach to enhance the effect of controllable generation. It can be universally applied to any diffusion model, allowing users to adjust the representation of specific image features in accordance with task requirements, thereby facilitating more precise and controllable generation outcomes. In particular, our designed experiments demonstrate that the FilterPrompt optimizes feature correlation, mitigates content conflicts during the generation process, and enhances the effect of controllable generation.