Visual Question Generation for Class Acquisition of Unknown Objects

TL;DR

This paper introduces a novel method for generating questions about unknown objects in images to facilitate learning new object classes, addressing the limitations of traditional recognition systems that only recognize known classes.

Contribution

It proposes a new framework combining object proposal, unknown object identification, and question generation modules for unknown object class acquisition.

Findings

Successfully generates questions about unknown objects

Human evaluation confirms effectiveness

Code and dataset publicly available

Abstract

Traditional image recognition methods only consider objects belonging to already learned classes. However, since training a recognition model with every object class in the world is unfeasible, a way of getting information on unknown objects (i.e., objects whose class has not been learned) is necessary. A way for an image recognition system to learn new classes could be asking a human about objects that are unknown. In this paper, we propose a method for generating questions about unknown objects in an image, as means to get information about classes that have not been learned. Our method consists of a module for proposing objects, a module for identifying unknown objects, and a module for generating questions about unknown objects. The experimental results via human evaluation show that our method can successfully get information about unknown objects in an image dataset. Our code and dataset are available at https://github.com/mil-tokyo/vqg-unknown.