PhyCustom: Towards Realistic Physical Customization in Text-to-Image Generation

TL;DR

This paper introduces PhyCustom, a fine-tuning framework with novel regularization losses that enables diffusion models to accurately incorporate physical concepts into text-to-image generation, improving realism and specificity.

Contribution

The paper presents a new fine-tuning method with isometric and decouple losses to enhance physical concept customization in diffusion-based image generation.

Findings

Outperforms previous methods in physical customization accuracy

Achieves better qualitative and quantitative results on diverse datasets

Effectively disentangles physical concepts during training

Abstract

Recent diffusion-based text-to-image customization methods have achieved significant success in understanding concrete concepts to control generation processes, such as styles and shapes. However, few efforts dive into the realistic yet challenging customization of physical concepts. The core limitation of current methods arises from the absence of explicitly introducing physical knowledge during training. Even when physics-related words appear in the input text prompts, our experiments consistently demonstrate that these methods fail to accurately reflect the corresponding physical properties in the generated results. In this paper, we propose PhyCustom, a fine-tuning framework comprising two novel regularization losses to activate diffusion model to perform physical customization. Specifically, the proposed isometric loss aims at activating diffusion models to learn physical concepts while decouple loss helps to eliminate the mixture learning of independent concepts. Experiments are conducted on a diverse dataset and our benchmark results demonstrate that PhyCustom outperforms previous state-of-the-art and popular methods in terms of physical customization quantitatively and qualitatively.