Visual Madlibs: Fill in the blank Image Generation and Question Answering

TL;DR

This paper introduces the Visual Madlibs dataset with targeted natural language descriptions for images, enabling new focused description and question-answering tasks, and demonstrates promising results with deep learning methods.

Contribution

The paper presents a large, targeted dataset for image description and inference, and applies it to novel description generation and question-answering tasks.

Findings

Promising results on description generation tasks

Effective use of joint-embedding and deep learning methods

Dataset facilitates targeted image understanding

Abstract

In this paper, we introduce a new dataset consisting of 360,001 focused natural language descriptions for 10,738 images. This dataset, the Visual Madlibs dataset, is collected using automatically produced fill-in-the-blank templates designed to gather targeted descriptions about: people and objects, their appearances, activities, and interactions, as well as inferences about the general scene or its broader context. We provide several analyses of the Visual Madlibs dataset and demonstrate its applicability to two new description generation tasks: focused description generation, and multiple-choice question-answering for images. Experiments using joint-embedding and deep learning methods show promising results on these tasks.