COPA: Efficient Vision-Language Pre-training Through Collaborative Object- and Patch-Text Alignment

TL;DR

This paper presents COPA, a vision-language pre-training method that efficiently integrates object information into ViT models through a novel patch-text alignment, significantly speeding up inference while maintaining high performance.

Contribution

It introduces a patch-text alignment mechanism that converts object signals into patch-level cues, enabling efficient and detailed cross-modal learning without expensive object detection.

Findings

Achieves 88% speedup over previous models.

Maintains or improves downstream task performance.

Uses only 5% annotated images for training.

Abstract

Vision-Language Pre-training (VLP) methods based on object detection enjoy the rich knowledge of fine-grained object-text alignment but at the cost of computationally expensive inference. Recent Visual-Transformer (ViT)-based approaches circumvent this issue while struggling with long visual sequences without detailed cross-modal alignment information. This paper introduces a ViT-based VLP technique that efficiently incorporates object information through a novel patch-text alignment mechanism. Specifically, we convert object-level signals into patch-level ones and devise a Patch-Text Alignment pre-training task (PTA) to learn a text-aware patch detector. By using off-the-shelf delicate object annotations in 5\% training images, we jointly train PTA with other conventional VLP objectives in an end-to-end manner, bypassing the high computational cost of object detection and yielding an effective patch detector that accurately detects text-relevant patches, thus considerably reducing patch sequences and accelerating computation within the ViT backbone. Our experiments on a variety of widely-used benchmarks reveal that our method achieves a speedup of nearly 88\% compared to prior VLP models while maintaining competitive or superior performance on downstream tasks with similar model size and data scale.