Expedited Training of Visual Conditioned Language Generation via Redundancy Reduction

TL;DR

This paper presents EVL_Gen, a fast and efficient framework for pre-training vision-language models that reduces training time by merging visual tokens gradually, achieving comparable performance with less data and enabling video adaptation.

Contribution

Introduces a one-stage, single-loss training framework that accelerates vision-language model pre-training by five times and reduces data requirements, while maintaining performance.

Findings

Training speed increased by a factor of 5.

Models perform well with only 10% of the data.

Framework adapts to video-conditioned tasks.

Abstract

In this paper, we introduce $\text{EVL}_{\text{Gen}}$, a streamlined framework designed for the pre-training of visually conditioned language generation models with high computational demands, utilizing frozen pre-trained large language models (LLMs). The conventional approach in vision-language pre-training (VLP) typically involves a two-stage optimization process: an initial resource-intensive phase dedicated to general-purpose vision-language representation learning, focused on extracting and consolidating relevant visual features. This is followed by a subsequent phase that emphasizes end-to-end alignment between visual and linguistic modalities. Our novel one-stage, single-loss framework bypasses the computationally demanding first training stage by gradually merging similar visual tokens during training, while avoiding model collapse caused by single-stage training of BLIP-2 type models. The gradual merging process effectively condenses visual information while preserving semantic richness, resulting in rapid convergence without compromising performance. Our experimental findings demonstrate that our approach accelerates the training of vision-language models by a factor of 5 without a noticeable impact on overall performance. Furthermore, we illustrate that our models significantly narrow the performance gap to current vision-language models using only 1/10 of the data. Finally, we showcase how our image-text models can seamlessly adapt to video-conditioned language generation tasks through novel soft attentive temporal token contextualizing modules. Code is available at \url{https://github.com/yiren-jian/EVLGen}.