Mining Contextualized Visual Associations from Images for Creativity Understanding

TL;DR

This paper presents a novel method for mining contextualized visual associations from images to generate creative, abstract captions, improving zero-shot image-text retrieval in creative domains and providing a new dataset for the community.

Contribution

It introduces a scalable approach to extract contextualized associations for visual elements, creating a large dataset of creative captions and enhancing vision-language models for creative understanding.

Findings

Generated 1.7 million creative captions for MSCOCO images.

Human evaluation confirms captions are visually grounded and increasingly abstract.

Fine-tuning on this dataset improves zero-shot retrieval in poetry and metaphor visualization.

Abstract

Understanding another person's creative output requires a shared language of association. However, when training vision-language models such as CLIP, we rely on web-scraped datasets containing short, predominantly literal, alt-text. In this work, we introduce a method for mining contextualized associations for salient visual elements in an image that can scale to any unlabeled dataset. Given an image, we can use these mined associations to generate high quality creative captions at increasing degrees of abstraction. With our method, we produce a new dataset of visual associations and 1.7m creative captions for the images in MSCOCO. Human evaluation confirms that these captions remain visually grounded while exhibiting recognizably increasing abstraction. Moreover, fine-tuning a visual encoder on this dataset yields meaningful improvements in zero-shot image-text retrieval in two creative domains: poetry and metaphor visualization. We release our dataset, our generation code and our models for use by the broader community.