Generating Compositional Color Representations from Text

TL;DR

This paper introduces a generative adversarial network that creates color representations from text phrases, especially attribute-object pairs, and demonstrates its effectiveness in image retrieval and classification tasks.

Contribution

It presents a novel GAN-based approach for compositional color generation from text, along with a new dataset creation pipeline and extensive ablation studies.

Findings

GAN achieves lower Frechet Inception Distance than discriminative models.

Generated color profiles improve image retrieval accuracy.

Method generalizes to other visual dimensions involving composition.

Abstract

We consider the cross-modal task of producing color representations for text phrases. Motivated by the fact that a significant fraction of user queries on an image search engine follow an (attribute, object) structure, we propose a generative adversarial network that generates color profiles for such bigrams. We design our pipeline to learn composition - the ability to combine seen attributes and objects to unseen pairs. We propose a novel dataset curation pipeline from existing public sources. We describe how a set of phrases of interest can be compiled using a graph propagation technique, and then mapped to images. While this dataset is specialized for our investigations on color, the method can be extended to other visual dimensions where composition is of interest. We provide detailed ablation studies that test the behavior of our GAN architecture with loss functions from the contrastive learning literature. We show that the generative model achieves lower Frechet Inception Distance than discriminative ones, and therefore predicts color profiles that better match those from real images. Finally, we demonstrate improved performance in image retrieval and classification, indicating the crucial role that color plays in these downstream tasks.