Gated Condition Injection without Multimodal Attention: Towards Controllable Linear-Attention Transformers

TL;DR

This paper introduces a new controllable diffusion framework using linear attention models like SANA, enabling efficient on-device image generation with improved flexibility and performance over existing methods.

Contribution

A novel gated conditioning module for linear attention diffusion models that supports multiple heterogeneous condition types effectively.

Findings

Achieves state-of-the-art controllable generation performance with linear-attention models.

Outperforms existing methods in fidelity and controllability.

Supports multiple condition types with a unified dual-path pipeline.

Abstract

Recent advances in diffusion-based controllable visual generation have led to remarkable improvements in image quality. However, these powerful models are typically deployed on cloud servers due to their large computational demands, raising serious concerns about user data privacy. To enable secure and efficient on-device generation, we explore in this paper controllable diffusion models built upon linear attention architectures, which offer superior scalability and efficiency, even on edge devices. Yet, our experiments reveal that existing controllable generation frameworks, such as ControlNet and OminiControl, either lack the flexibility to support multiple heterogeneous condition types or suffer from slow convergence on such linear-attention models. To address these limitations, we propose a novel controllable diffusion framework tailored for linear attention backbones like SANA. The core of our method lies in a unified gated conditioning module working in a dual-path pipeline, which effectively integrates multi-type conditional inputs, such as spatially aligned and non-aligned cues. Extensive experiments on multiple tasks and benchmarks demonstrate that our approach achieves state-of-the-art controllable generation performance based on linear-attention models, surpassing existing methods in terms of fidelity and controllability.