OptiPrune: Boosting Prompt-Image Consistency with Attention-Guided Noise and Dynamic Token Selection

TL;DR

OptiPrune is a novel framework that improves semantic alignment in text-to-image diffusion models by combining attention-guided noise optimization with efficient token pruning, achieving high quality with less computation.

Contribution

It introduces a unified approach integrating distribution-aware noise optimization and similarity-based token pruning for better efficiency and semantic fidelity.

Findings

Achieves state-of-the-art prompt-image consistency on benchmark datasets.

Reduces computational cost significantly compared to existing methods.

Maintains high semantic alignment without sacrificing inference efficiency.

Abstract

Text-to-image diffusion models often struggle to achieve accurate semantic alignment between generated images and text prompts while maintaining efficiency for deployment on resource-constrained hardware. Existing approaches either incur substantial computational overhead through noise optimization or compromise semantic fidelity by aggressively pruning tokens. In this work, we propose OptiPrune, a unified framework that combines distribution-aware initial noise optimization with similarity-based token pruning to address both challenges simultaneously. Specifically, (1) we introduce a distribution-aware noise optimization module guided by attention scores to steer the initial latent noise toward semantically meaningful regions, mitigating issues such as subject neglect and feature entanglement; (2) we design a hardware-efficient token pruning strategy that selects representative base tokens via patch-wise similarity, injects randomness to enhance generalization, and recovers pruned tokens using maximum similarity copying before attention operations. Our method preserves the Gaussian prior during noise optimization and enables efficient inference without sacrificing alignment quality. Experiments on benchmark datasets, including Animal-Animal, demonstrate that OptiPrune achieves state-of-the-art prompt-image consistency with significantly reduced computational cost.