UFO: A UniFied TransfOrmer for Vision-Language Representation Learning

TL;DR

UFO introduces a unified transformer model capable of handling unimodal and multimodal vision-language tasks through multi-task pre-training, simplifying architecture and achieving state-of-the-art results across various benchmarks.

Contribution

The paper presents a single transformer framework for all vision-language tasks, reducing complexity and improving performance via multi-task learning during pre-training.

Findings

Achieves new state-of-the-art on visual question answering.

Sets new records on COCO image captioning and nocaps.

Performs competitively on image-text retrieval.

Abstract

In this paper, we propose a single UniFied transfOrmer (UFO), which is capable of processing either unimodal inputs (e.g., image or language) or multimodal inputs (e.g., the concatenation of the image and the question), for vision-language (VL) representation learning. Existing approaches typically design an individual network for each modality and/or a specific fusion network for multimodal tasks. To simplify the network architecture, we use a single transformer network and enforce multi-task learning during VL pre-training, which includes the image-text contrastive loss, image-text matching loss, and masked language modeling loss based on the bidirectional and the seq2seq attention mask. The same transformer network is used as the image encoder, the text encoder, or the fusion network in different pre-training tasks. Empirically, we observe less conflict among different tasks and achieve new state of the arts on visual question answering, COCO image captioning (cross-entropy optimization) and nocaps (in SPICE). On other downstream tasks, e.g., image-text retrieval, we also achieve competitive performance.