Large-scale Pretraining for Visual Dialog: A Simple State-of-the-Art Baseline

TL;DR

This paper introduces a simple yet effective pretraining approach for visual dialog using vision-language datasets, significantly improving performance on standard metrics compared to prior methods.

Contribution

It adapts the ViLBERT model for visual dialog, demonstrating the benefits of pretraining on related datasets and analyzing the impact of dense annotations on evaluation metrics.

Findings

Pretraining on Conceptual Captions and VQA improves visual dialog performance.

Finetuning with dense annotations boosts NDCG but reduces MRR.

Trade-off observed between NDCG and MRR metrics due to annotation types.

Abstract

Prior work in visual dialog has focused on training deep neural models on VisDial in isolation. Instead, we present an approach to leverage pretraining on related vision-language datasets before transferring to visual dialog. We adapt the recently proposed ViLBERT (Lu et al., 2019) model for multi-turn visually-grounded conversations. Our model is pretrained on the Conceptual Captions and Visual Question Answering datasets, and finetuned on VisDial. Our best single model outperforms prior published work (including model ensembles) by more than 1% absolute on NDCG and MRR. Next, we find that additional finetuning using "dense" annotations in VisDial leads to even higher NDCG -- more than 10% over our base model -- but hurts MRR -- more than 17% below our base model! This highlights a trade-off between the two primary metrics -- NDCG and MRR -- which we find is due to dense annotations not correlating well with the original ground-truth answers to questions.