Unveiling the Dreams of Word Embeddings: Towards Language-Driven Image Generation

TL;DR

This paper presents a method for generating images from word embeddings by mapping semantic vectors to visual space and then to pixels, capturing general visual features and categories.

Contribution

It introduces a simple two-step mapping approach from word embeddings to images, demonstrating the feasibility of language-driven image generation.

Findings

Generated images reflect semantic content and visual properties.

System can distinguish between broad object categories.

User studies confirm the meaningfulness of generated images.

Abstract

We introduce language-driven image generation, the task of generating an image visualizing the semantic contents of a word embedding, e.g., given the word embedding of grasshopper, we generate a natural image of a grasshopper. We implement a simple method based on two mapping functions. The first takes as input a word embedding (as produced, e.g., by the word2vec toolkit) and maps it onto a high-level visual space (e.g., the space defined by one of the top layers of a Convolutional Neural Network). The second function maps this abstract visual representation to pixel space, in order to generate the target image. Several user studies suggest that the current system produces images that capture general visual properties of the concepts encoded in the word embedding, such as color or typical environment, and are sufficient to discriminate between general categories of objects.