VDialogUE: A Unified Evaluation Benchmark for Visually-grounded Dialogue

TL;DR

VDialogUE introduces a comprehensive benchmark and evaluation metric for visually-grounded dialogue systems, covering multiple tasks and datasets to standardize assessment and foster progress in the field.

Contribution

It proposes a unified evaluation benchmark with five core tasks, a novel VDscore metric, and a baseline model to advance multi-modal dialogue research.

Findings

VDialogUE covers six datasets and five tasks for evaluation.

VDscore provides a comprehensive performance assessment.

Baseline model VISIT demonstrates effective multi-modal dialogue capabilities.

Abstract

Visually-grounded dialog systems, which integrate multiple modes of communication such as text and visual inputs, have become an increasingly popular area of investigation. However, the absence of a standardized evaluation framework poses a challenge in assessing the development of this field. To this end, we propose \textbf{VDialogUE}, a \textbf{V}isually-grounded \textbf{Dialog}ue benchmark for \textbf{U}nified \textbf{E}valuation. It defines five core multi-modal dialogue tasks and covers six datasets. Furthermore, in order to provide a comprehensive assessment of the model's performance across all tasks, we developed a novel evaluation metric called VDscore, which is based on the Analytic Hierarchy Process~(AHP) method. Additionally, we present a straightforward yet efficient baseline model, named \textbf{VISIT}~(\textbf{VIS}ually-grounded d\textbf{I}alog \textbf{T}ransformer), to promote the advancement of general multi-modal dialogue systems. It progressively builds its multi-modal foundation and dialogue capability via a two-stage pre-training strategy. We believe that the VDialogUE benchmark, along with the evaluation scripts and our baseline models, will accelerate the development of visually-grounded dialog systems and lead to the development of more sophisticated and effective pre-trained models.