Efficient Text-Guided Convolutional Adapter for the Diffusion Model

TL;DR

The paper presents Nexus Adapters, efficient prompt-guided adapters for diffusion models that improve structure-preserving conditional image generation with fewer parameters and enhanced multimodal understanding.

Contribution

Introduction of Nexus Prime and Slim adapters that are prompt-guided, multimodal, and parameter-efficient, improving structure-preserving diffusion-based image generation.

Findings

Nexus Prime significantly improves performance with only 8M extra parameters.

Nexus Slim achieves state-of-the-art results with 18M fewer parameters.

Adapters effectively incorporate prompt and structural input understanding.

Abstract

We introduce the Nexus Adapters, novel text-guided efficient adapters to the diffusion-based framework for the Structure Preserving Conditional Generation (SPCG). Recently, structure-preserving methods have achieved promising results in conditional image generation by using a base model for prompt conditioning and an adapter for structure input, such as sketches or depth maps. These approaches are highly inefficient and sometimes require equal parameters in the adapter compared to the base architecture. It is not always possible to train the model since the diffusion model is itself costly, and doubling the parameter is highly inefficient. In these approaches, the adapter is not aware of the input prompt; therefore, it is optimal only for the structural input but not for the input prompt. To overcome the above challenges, we proposed two efficient adapters, Nexus Prime and Slim, which are guided by prompts and structural inputs. Each Nexus Block incorporates cross-attention mechanisms to enable rich multimodal conditioning. Therefore, the proposed adapter has a better understanding of the input prompt while preserving the structure. We conducted extensive experiments on the proposed models and demonstrated that the Nexus Prime adapter significantly enhances performance, requiring only 8M additional parameters compared to the baseline, T2I-Adapter. Furthermore, we also introduced a lightweight Nexus Slim adapter with 18M fewer parameters than the T2I-Adapter, which still achieved state-of-the-art results. Code: https://github.com/arya-domain/Nexus-Adapters