LXMERT: Learning Cross-Modality Encoder Representations from Transformers

TL;DR

LXMERT introduces a large-scale Transformer-based framework that learns cross-modal representations from image and text data, achieving state-of-the-art results in visual question answering and reasoning tasks through extensive pre-training and fine-tuning.

Contribution

The paper presents a novel multi-encoder Transformer model with diverse pre-training tasks for vision-and-language understanding, advancing cross-modal reasoning capabilities.

Findings

Achieves state-of-the-art on VQA and GQA datasets.

Improves NLVR2 accuracy by 22% absolute.

Demonstrates effectiveness of pre-training and model components.

Abstract

Vision-and-language reasoning requires an understanding of visual concepts, language semantics, and, most importantly, the alignment and relationships between these two modalities. We thus propose the LXMERT (Learning Cross-Modality Encoder Representations from Transformers) framework to learn these vision-and-language connections. In LXMERT, we build a large-scale Transformer model that consists of three encoders: an object relationship encoder, a language encoder, and a cross-modality encoder. Next, to endow our model with the capability of connecting vision and language semantics, we pre-train the model with large amounts of image-and-sentence pairs, via five diverse representative pre-training tasks: masked language modeling, masked object prediction (feature regression and label classification), cross-modality matching, and image question answering. These tasks help in learning both intra-modality and cross-modality relationships. After fine-tuning from our pre-trained parameters, our model achieves the state-of-the-art results on two visual question answering datasets (i.e., VQA and GQA). We also show the generalizability of our pre-trained cross-modality model by adapting it to a challenging visual-reasoning task, NLVR2, and improve the previous best result by 22% absolute (54% to 76%). Lastly, we demonstrate detailed ablation studies to prove that both our novel model components and pre-training strategies significantly contribute to our strong results; and also present several attention visualizations for the different encoders. Code and pre-trained models publicly available at: https://github.com/airsplay/lxmert