Learning by Asking Questions

TL;DR

This paper proposes a learning-by-asking framework for visual question answering, where the model actively asks questions to learn efficiently, mimicking natural learning, and demonstrates improved data efficiency and generalization on CLEVR.

Contribution

Introduces an interactive learning framework for VQA that enables models to ask questions, discovering curricula and improving sample efficiency compared to traditional methods.

Findings

LBA discovers an easy-to-hard curriculum during learning.

LBA-generated data matches or exceeds the quality of training data.

The model's questions generalize to other VQA models and new distributions.

Abstract

We introduce an interactive learning framework for the development and testing of intelligent visual systems, called learning-by-asking (LBA). We explore LBA in context of the Visual Question Answering (VQA) task. LBA differs from standard VQA training in that most questions are not observed during training time, and the learner must ask questions it wants answers to. Thus, LBA more closely mimics natural learning and has the potential to be more data-efficient than the traditional VQA setting. We present a model that performs LBA on the CLEVR dataset, and show that it automatically discovers an easy-to-hard curriculum when learning interactively from an oracle. Our LBA generated data consistently matches or outperforms the CLEVR train data and is more sample efficient. We also show that our model asks questions that generalize to state-of-the-art VQA models and to novel test time distributions.