Learning by Asking Questions for Knowledge-based Novel Object Recognition

TL;DR

This paper introduces a framework where a question generation system helps a classifier recognize novel objects by acquiring external knowledge, inspired by human questioning behavior, improving recognition of unseen classes.

Contribution

It proposes a novel pipeline combining knowledge-based object recognition with question generation to enable recognition of novel objects in real-world scenarios.

Findings

Effective acquisition of knowledge about novel objects.

Improved recognition accuracy over baseline methods.

A new dataset for training knowledge-aware question generation.

Abstract

In real-world object recognition, there are numerous object classes to be recognized. Conventional image recognition based on supervised learning can only recognize object classes that exist in the training data, and thus has limited applicability in the real world. On the other hand, humans can recognize novel objects by asking questions and acquiring knowledge about them. Inspired by this, we study a framework for acquiring external knowledge through question generation that would help the model instantly recognize novel objects. Our pipeline consists of two components: the Object Classifier, which performs knowledge-based object recognition, and the Question Generator, which generates knowledge-aware questions to acquire novel knowledge. We also propose a question generation strategy based on the confidence of the knowledge-aware prediction of the Object Classifier. To train the Question Generator, we construct a dataset that contains knowledge-aware questions about objects in the images. Our experiments show that the proposed pipeline effectively acquires knowledge about novel objects compared to several baselines.