Audio Visual Scene-Aware Dialog (AVSD) Challenge at DSTC7

TL;DR

The paper introduces the AVSD challenge and dataset to advance scene-aware dialog systems capable of discussing objects and events in videos, integrating multiple AI research areas.

Contribution

It presents a new challenge and dataset for developing dialog systems that understand and describe video content in conversational settings.

Findings

First AVSD challenge dataset released

Baseline models established for AVSD task

Encourages multi-disciplinary research in video-based dialog systems

Abstract

Scene-aware dialog systems will be able to have conversations with users about the objects and events around them. Progress on such systems can be made by integrating state-of-the-art technologies from multiple research areas including end-to-end dialog systems visual dialog, and video description. We introduce the Audio Visual Scene Aware Dialog (AVSD) challenge and dataset. In this challenge, which is one track of the 7th Dialog System Technology Challenges (DSTC7) workshop1, the task is to build a system that generates responses in a dialog about an input video