Generating Natural Questions from Images for Multimodal Assistants

TL;DR

This paper introduces a new dataset and an approach for generating diverse, meaningful questions from images, tailored for multimodal assistants, and achieves state-of-the-art results in question generation quality.

Contribution

It provides a human-annotated dataset of questions for multimodal assistants and an automatic question generation method considering image content and metadata.

Findings

Achieved state-of-the-art results on question generation metrics.

Generated questions are relevant, diverse, and human-like.

The approach effectively incorporates image metadata for better questions.

Abstract

Generating natural, diverse, and meaningful questions from images is an essential task for multimodal assistants as it confirms whether they have understood the object and scene in the images properly. The research in visual question answering (VQA) and visual question generation (VQG) is a great step. However, this research does not capture questions that a visually-abled person would ask multimodal assistants. Recently published datasets such as KB-VQA, FVQA, and OK-VQA try to collect questions that look for external knowledge which makes them appropriate for multimodal assistants. However, they still contain many obvious and common-sense questions that humans would not usually ask a digital assistant. In this paper, we provide a new benchmark dataset that contains questions generated by human annotators keeping in mind what they would ask multimodal digital assistants. Large scale annotations for several hundred thousand images are expensive and time-consuming, so we also present an effective way of automatically generating questions from unseen images. In this paper, we present an approach for generating diverse and meaningful questions that consider image content and metadata of image (e.g., location, associated keyword). We evaluate our approach using standard evaluation metrics such as BLEU, METEOR, ROUGE, and CIDEr to show the relevance of generated questions with human-provided questions. We also measure the diversity of generated questions using generative strength and inventiveness metrics. We report new state-of-the-art results on the public and our datasets.