Vision Model Pre-training on Interleaved Image-Text Data via Latent Compression Learning

TL;DR

This paper introduces Latent Compression Learning (LCL), a novel pre-training method for vision models that effectively exploits interleaved image-text data by maximizing mutual information, leading to robust visual representations.

Contribution

The paper presents a new pre-training approach called LCL that leverages interleaved image-text data using latent compression learning, a concept inspired by NLP success.

Findings

LCL matches CLIP performance on paired datasets.

LCL effectively learns from interleaved image-text data.

The method demonstrates robustness in visual representation learning.

Abstract

Recently, vision model pre-training has evolved from relying on manually annotated datasets to leveraging large-scale, web-crawled image-text data. Despite these advances, there is no pre-training method that effectively exploits the interleaved image-text data, which is very prevalent on the Internet. Inspired by the recent success of compression learning in natural language processing, we propose a novel vision model pre-training method called Latent Compression Learning (LCL) for interleaved image-text data. This method performs latent compression learning by maximizing the mutual information between the inputs and outputs of a causal attention model. The training objective can be decomposed into two basic tasks: 1) contrastive learning between visual representation and preceding context, and 2) generating subsequent text based on visual representation. Our experiments demonstrate that our method not only matches the performance of CLIP on paired pre-training datasets (e.g., LAION), but can also leverage interleaved pre-training data (e.g., MMC4) to learn robust visual representation from scratch, showcasing the potential of vision model pre-training with interleaved image-text data. Code is released at https://github.com/OpenGVLab/LCL.