Are we pretraining it right? Digging deeper into visio-linguistic pretraining

TL;DR

This paper investigates how the choice of pretraining datasets impacts the performance of visio-linguistic models, revealing that domain similarity and data generation methods significantly influence downstream task results.

Contribution

It systematically studies the effects of dataset domain similarity and data generation on pretraining effectiveness, providing insights for better dataset selection.

Findings

Automatically generated data closer to downstream domain improves performance.

Some reasonable dataset choices are ineffective for certain tasks.

Simple pretraining design choices can achieve near state-of-the-art results.

Abstract

Numerous recent works have proposed pretraining generic visio-linguistic representations and then finetuning them for downstream vision and language tasks. While architecture and objective function design choices have received attention, the choice of pretraining datasets has received little attention. In this work, we question some of the default choices made in literature. For instance, we systematically study how varying similarity between the pretraining dataset domain (textual and visual) and the downstream domain affects performance. Surprisingly, we show that automatically generated data in a domain closer to the downstream task (e.g., VQA v2) is a better choice for pretraining than "natural" data but of a slightly different domain (e.g., Conceptual Captions). On the other hand, some seemingly reasonable choices of pretraining datasets were found to be entirely ineffective for some downstream tasks. This suggests that despite the numerous recent efforts, vision & language pretraining does not quite work "out of the box" yet. Overall, as a by-product of our study, we find that simple design choices in pretraining can help us achieve close to state-of-art results on downstream tasks without any architectural changes.