Bootstrapping Vision-Language Learning with Decoupled Language Pre-training

TL;DR

This paper introduces a novel prompt prediction approach for vision-language pre-training that improves performance and reduces data requirements by focusing on language prompts, trained solely on linguistic data.

Contribution

It proposes the Prompt-Transformer (P-Former), a language-only trained model that predicts optimal prompts to align visual features with text, enhancing VL pre-training efficiency.

Findings

Significantly improves image-to-text baseline performance.

Reduces performance gap between models trained on different data scales.

Demonstrates modality-agnostic applicability, including video learning.

Abstract

We present a novel methodology aimed at optimizing the application of frozen large language models (LLMs) for resource-intensive vision-language (VL) pre-training. The current paradigm uses visual features as prompts to guide language models, with a focus on determining the most relevant visual features for corresponding text. Our approach diverges by concentrating on the language component, specifically identifying the optimal prompts to align with visual features. We introduce the Prompt-Transformer (P-Former), a model that predicts these ideal prompts, which is trained exclusively on linguistic data, bypassing the need for image-text pairings. This strategy subtly bifurcates the end-to-end VL training process into an additional, separate stage. Our experiments reveal that our framework significantly enhances the performance of a robust image-to-text baseline (BLIP-2), and effectively narrows the performance gap between models trained with either 4M or 129M image-text pairs. Importantly, our framework is modality-agnostic and flexible in terms of architectural design, as validated by its successful application in a video learning task using varied base modules. The code will be made available at https://github.com/yiren-jian/BLIText.