SSD: Towards Better Text-Image Consistency Metric in Text-to-Image Generation

TL;DR

This paper introduces SSD, a new CLIP-based metric for better measuring text-image consistency in generation, and proposes PDF-GAN, a model that improves semantic alignment between text and images.

Contribution

The paper develops a novel distributionally-founded CLIP-based metric SSD and a new GAN architecture PDF-GAN with plug-and-play components for enhanced text-image semantic consistency.

Findings

SSD correlates better with human judgment of consistency

PDF-GAN achieves superior text-image alignment on benchmark datasets

Proposed methods outperform existing state-of-the-art in consistency metrics

Abstract

Generating consistent and high-quality images from given texts is essential for visual-language understanding. Although impressive results have been achieved in generating high-quality images, text-image consistency is still a major concern in existing GAN-based methods. Particularly, the most popular metric $R$-precision may not accurately reflect the text-image consistency, often resulting in very misleading semantics in the generated images. Albeit its significance, how to design a better text-image consistency metric surprisingly remains under-explored in the community. In this paper, we make a further step forward to develop a novel CLIP-based metric termed as Semantic Similarity Distance ($SSD$), which is both theoretically founded from a distributional viewpoint and empirically verified on benchmark datasets. Benefiting from the proposed metric, we further design the Parallel Deep Fusion Generative Adversarial Networks (PDF-GAN) that aims at improving text-image consistency by fusing semantic information at different granularities and capturing accurate semantics. Equipped with two novel plug-and-play components: Hard-Negative Sentence Constructor and Semantic Projection, the proposed PDF-GAN can mitigate inconsistent semantics and bridge the text-image semantic gap. A series of experiments show that, as opposed to current state-of-the-art methods, our PDF-GAN can lead to significantly better text-image consistency while maintaining decent image quality on the CUB and COCO datasets.