Learning More May Not Be Better: Knowledge Transferability in Vision and Language Tasks

TL;DR

This paper investigates the complexities of knowledge transferability in vision-and-language models, revealing that more data and shared tasks do not always lead to improved performance, with transfer success depending on various factors.

Contribution

It provides an exhaustive analysis of knowledge transfer in multi-modal tasks, highlighting conditions where transfer is beneficial or detrimental.

Findings

Tasks in the same group often improve each other

Dataset size influences transfer effectiveness

Pre-training stage impacts transfer success

Abstract

Is more data always better to train vision-and-language models? We study knowledge transferability in multi-modal tasks. The current tendency in machine learning is to assume that by joining multiple datasets from different tasks their overall performance will improve. However, we show that not all the knowledge transfers well or has a positive impact on related tasks, even when they share a common goal. We conduct an exhaustive analysis based on hundreds of cross-experiments on 12 vision-and-language tasks categorized in 4 groups. Whereas tasks in the same group are prone to improve each other, results show that this is not always the case. Other factors such as dataset size or pre-training stage have also a great impact on how well the knowledge is transferred.