Describing Unseen Videos via Multi-Modal Cooperative Dialog Agents

TL;DR

This paper introduces a novel multi-modal cooperative dialog task where one agent describes an unseen video based on limited static frames and dialog, with a focus on knowledge transfer and improved video description.

Contribution

It proposes a new task and a QA-Cooperative Network with dynamic dialog update, enabling one agent to effectively describe unseen videos through cooperative learning.

Findings

Q-BOT effectively learns to describe unseen videos.

The model achieves promising performance with full dialog history.

Cooperative learning improves video description accuracy.

Abstract

With the arising concerns for the AI systems provided with direct access to abundant sensitive information, researchers seek to develop more reliable AI with implicit information sources. To this end, in this paper, we introduce a new task called video description via two multi-modal cooperative dialog agents, whose ultimate goal is for one conversational agent to describe an unseen video based on the dialog and two static frames. Specifically, one of the intelligent agents - Q-BOT - is given two static frames from the beginning and the end of the video, as well as a finite number of opportunities to ask relevant natural language questions before describing the unseen video. A-BOT, the other agent who has already seen the entire video, assists Q-BOT to accomplish the goal by providing answers to those questions. We propose a QA-Cooperative Network with a dynamic dialog history update learning mechanism to transfer knowledge from A-BOT to Q-BOT, thus helping Q-BOT to better describe the video. Extensive experiments demonstrate that Q-BOT can effectively learn to describe an unseen video by the proposed model and the cooperative learning method, achieving the promising performance where Q-BOT is given the full ground truth history dialog.