Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers

TL;DR

Pixel-BERT introduces a unified deep multi-modal transformer that aligns image pixels with text directly, improving performance on vision-language tasks without relying on region-based features or bounding box annotations.

Contribution

It presents a novel end-to-end model that aligns pixels with text, overcoming limitations of region-based features and reducing annotation costs.

Findings

Achieves state-of-the-art results on VQA, image-text retrieval, and NLVR tasks.

Boosts VQA performance by 2.17 points over previous SOTA.

Uses a novel pixel sampling mechanism and multi-task pre-training.

Abstract

We propose Pixel-BERT to align image pixels with text by deep multi-modal transformers that jointly learn visual and language embedding in a unified end-to-end framework. We aim to build a more accurate and thorough connection between image pixels and language semantics directly from image and sentence pairs instead of using region-based image features as the most recent vision and language tasks. Our Pixel-BERT which aligns semantic connection in pixel and text level solves the limitation of task-specific visual representation for vision and language tasks. It also relieves the cost of bounding box annotations and overcomes the unbalance between semantic labels in visual task and language semantic. To provide a better representation for down-stream tasks, we pre-train a universal end-to-end model with image and sentence pairs from Visual Genome dataset and MS-COCO dataset. We propose to use a random pixel sampling mechanism to enhance the robustness of visual representation and to apply the Masked Language Model and Image-Text Matching as pre-training tasks. Extensive experiments on downstream tasks with our pre-trained model show that our approach makes the most state-of-the-arts in downstream tasks, including Visual Question Answering (VQA), image-text retrieval, Natural Language for Visual Reasoning for Real (NLVR). Particularly, we boost the performance of a single model in VQA task by 2.17 points compared with SOTA under fair comparison.