RLIPv2: Fast Scaling of Relational Language-Image Pre-training

TL;DR

RLIPv2 introduces a fast-converging relational pre-training model that scales effectively to large pseudo-labelled scene graph data, significantly improving relational reasoning in vision tasks with state-of-the-art results.

Contribution

The paper presents RLIPv2, a novel model with ALIF for faster training and a method to generate large-scale relational data, enabling scalable and efficient relational pre-training.

Findings

Achieves state-of-the-art on Human-Object Interaction Detection benchmarks.

Yields high performance with minimal data, e.g., 32.22mAP with 1% data.

Demonstrates fast convergence and superior relational reasoning capabilities.

Abstract

Relational Language-Image Pre-training (RLIP) aims to align vision representations with relational texts, thereby advancing the capability of relational reasoning in computer vision tasks. However, hindered by the slow convergence of RLIPv1 architecture and the limited availability of existing scene graph data, scaling RLIPv1 is challenging. In this paper, we propose RLIPv2, a fast converging model that enables the scaling of relational pre-training to large-scale pseudo-labelled scene graph data. To enable fast scaling, RLIPv2 introduces Asymmetric Language-Image Fusion (ALIF), a mechanism that facilitates earlier and deeper gated cross-modal fusion with sparsified language encoding layers. ALIF leads to comparable or better performance than RLIPv1 in a fraction of the time for pre-training and fine-tuning. To obtain scene graph data at scale, we extend object detection datasets with free-form relation labels by introducing a captioner (e.g., BLIP) and a designed Relation Tagger. The Relation Tagger assigns BLIP-generated relation texts to region pairs, thus enabling larger-scale relational pre-training. Through extensive experiments conducted on Human-Object Interaction Detection and Scene Graph Generation, RLIPv2 shows state-of-the-art performance on three benchmarks under fully-finetuning, few-shot and zero-shot settings. Notably, the largest RLIPv2 achieves 23.29mAP on HICO-DET without any fine-tuning, yields 32.22mAP with just 1% data and yields 45.09mAP with 100% data. Code and models are publicly available at https://github.com/JacobYuan7/RLIPv2.