Dynamic Prompting of Frozen Text-to-Image Diffusion Models for Panoptic Narrative Grounding

TL;DR

This paper introduces a novel dynamic prompting method for frozen text-to-image diffusion models, significantly improving panoptic narrative grounding by enhancing fine-grained image-text alignment and segmentation accuracy.

Contribution

It proposes the Extractive-Injective Phrase Adapter and Multi-Level Mutual Aggregation modules to dynamically update prompts and fuse features, advancing PNG performance.

Findings

Achieves state-of-the-art results on PNG benchmark.

Demonstrates improved fine-grained image-text alignment.

Enhances segmentation accuracy with dynamic prompts.

Abstract

Panoptic narrative grounding (PNG), whose core target is fine-grained image-text alignment, requires a panoptic segmentation of referred objects given a narrative caption. Previous discriminative methods achieve only weak or coarse-grained alignment by panoptic segmentation pretraining or CLIP model adaptation. Given the recent progress of text-to-image Diffusion models, several works have shown their capability to achieve fine-grained image-text alignment through cross-attention maps and improved general segmentation performance. However, the direct use of phrase features as static prompts to apply frozen Diffusion models to the PNG task still suffers from a large task gap and insufficient vision-language interaction, yielding inferior performance. Therefore, we propose an Extractive-Injective Phrase Adapter (EIPA) bypass within the Diffusion UNet to dynamically update phrase prompts with image features and inject the multimodal cues back, which leverages the fine-grained image-text alignment capability of Diffusion models more sufficiently. In addition, we also design a Multi-Level Mutual Aggregation (MLMA) module to reciprocally fuse multi-level image and phrase features for segmentation refinement. Extensive experiments on the PNG benchmark show that our method achieves new state-of-the-art performance.