Synthesizing Novel Pairs of Image and Text

TL;DR

This paper introduces methods for generating new, paired image and text data by leveraging GANs and sequence models, enabling cross-domain and cycle-consistent synthesis for improved multimodal data creation.

Contribution

The paper proposes novel strategies for synthesizing image-text pairs across multiple domains using GANs, sequence models, and cycle-consistency techniques.

Findings

Effective generation of novel image-caption pairs

Cross-domain synthesis capabilities demonstrated

Cycle-consistency enhances data quality

Abstract

Generating novel pairs of image and text is a problem that combines computer vision and natural language processing. In this paper, we present strategies for generating novel image and caption pairs based on existing captioning datasets. The model takes advantage of recent advances in generative adversarial networks and sequence-to-sequence modeling. We make generalizations to generate paired samples from multiple domains. Furthermore, we study cycles -- generating from image to text then back to image and vise versa, as well as its connection with autoencoders.