LightCLIP: Learning Multi-Level Interaction for Lightweight Vision-Language Models

TL;DR

This paper introduces LightCLIP, a lightweight vision-language model that employs multi-level interaction and refined alignment objectives to improve performance on downstream tasks without extra inference cost.

Contribution

The paper proposes a multi-level interaction paradigm, including a relaxed bipartite matching for token-level alignment and an MLM objective, to enhance lightweight CLIP models.

Findings

Achieves higher downstream task performance without extra inference cost.

Improves fine-grained image-text alignment with relaxed bipartite matching.

Leverages MLM with an auxiliary fusion module to maximize text encoder potential.

Abstract

Vision-language pre-training like CLIP has shown promising performance on various downstream tasks such as zero-shot image classification and image-text retrieval. Most of the existing CLIP-alike works usually adopt relatively large image encoders like ResNet50 and ViT, while the lightweight counterparts are rarely discussed. In this paper, we propose a multi-level interaction paradigm for training lightweight CLIP models. Firstly, to mitigate the problem that some image-text pairs are not strictly one-to-one correspondence, we improve the conventional global instance-level alignment objective by softening the label of negative samples progressively. Secondly, a relaxed bipartite matching based token-level alignment objective is introduced for finer-grained alignment between image patches and textual words. Moreover, based on the observation that the accuracy of CLIP model does not increase correspondingly as the parameters of text encoder increase, an extra objective of masked language modeling (MLM) is leveraged for maximizing the potential of the shortened text encoder. In practice, an auxiliary fusion module injecting unmasked image embedding into masked text embedding at different network stages is proposed for enhancing the MLM. Extensive experiments show that without introducing additional computational cost during inference, the proposed method achieves a higher performance on multiple downstream tasks.