Evaluating the Capability of Video Question Generation for Expert Knowledge Elicitation

TL;DR

This paper introduces a new evaluation protocol and dataset for assessing video question generation models based on their ability to elicit expert knowledge, emphasizing question quality over answerability.

Contribution

We propose a novel evaluation protocol and create the EgoExoAsk dataset to better measure question quality in video question generation for expert knowledge elicitation.

Findings

Evaluation protocol aligns with question generation settings

Models with richer context perform better in the evaluation

EgoExoAsk dataset supports effective benchmarking

Abstract

Skilled human interviewers can extract valuable information from experts. This raises a fundamental question: what makes some questions more effective than others? To address this, a quantitative evaluation of question-generation models is essential. Video question generation (VQG) is a topic for video question answering (VideoQA), where questions are generated for given answers. Their evaluation typically focuses on the ability to answer questions, rather than the quality of generated questions. In contrast, we focus on the question quality in eliciting unseen knowledge from human experts. For a continuous improvement of VQG models, we propose a protocol that evaluates the ability by simulating question-answering communication with experts using a question-to-answer retrieval. We obtain the retriever by constructing a novel dataset, EgoExoAsk, which comprises 27,666 QA pairs generated from Ego-Exo4D's expert commentary annotation. The EgoExoAsk training set is used to obtain the retriever, and the benchmark is constructed on the validation set with Ego-Exo4D video segments. Experimental results demonstrate our metric reasonably aligns with question generation settings: models accessing richer context are evaluated better, supporting that our protocol works as intended. The EgoExoAsk dataset is available in https://github.com/omron-sinicx/VQG4ExpertKnowledge .