Condition Weaving Meets Expert Modulation: Towards Universal and Controllable Image Generation

TL;DR

This paper introduces UniGen, a unified framework for controllable image generation that reduces redundancy and enhances efficiency by integrating diverse conditions through novel modules like CoMoE and WeaveNet.

Contribution

The paper proposes CoMoE and WeaveNet modules to unify conditional image generation, reducing model redundancy and improving interaction between control signals.

Findings

Achieves state-of-the-art performance on multiple datasets.

Effectively mitigates feature entanglement and redundancy.

Enhances versatility and efficiency in controllable image generation.

Abstract

The image-to-image generation task aims to produce controllable images by leveraging conditional inputs and prompt instructions. However, existing methods often train separate control branches for each type of condition, leading to redundant model structures and inefficient use of computational resources. To address this, we propose a Unified image-to-image Generation (UniGen) framework that supports diverse conditional inputs while enhancing generation efficiency and expressiveness. Specifically, to tackle the widely existing parameter redundancy and computational inefficiency in controllable conditional generation architectures, we propose the Condition Modulated Expert (CoMoE) module. This module aggregates semantically similar patch features and assigns them to dedicated expert modules for visual representation and conditional modeling. By enabling independent modeling of foreground features under different conditions, CoMoE effectively mitigates feature entanglement and redundant computation in multi-condition scenarios. Furthermore, to bridge the information gap between the backbone and control branches, we propose WeaveNet, a dynamic, snake-like connection mechanism that enables effective interaction between global text-level control from the backbone and fine-grained control from conditional branches. Extensive experiments on the Subjects-200K and MultiGen-20M datasets across various conditional image generation tasks demonstrate that our method consistently achieves state-of-the-art performance, validating its advantages in both versatility and effectiveness. The code has been uploaded to https://github.com/gavin-gqzhang/UniGen.