Towards Cross-modal Backward-compatible Representation Learning for Vision-Language Models

TL;DR

This paper introduces a novel cross-modal backward-compatible training method for vision-language models, enabling efficient model upgrades without re-computing large datasets, by using a projection module pretrained with text data.

Contribution

It extends vision-only backward-compatible training to cross-modal retrieval, proposing a projection module that aligns new and old model embeddings efficiently without using the old model during training.

Findings

Effective cross-modal backward compatibility achieved.

Reduces data and computational requirements for model upgrades.

Enables backfill-free upgrades for vision-language models.

Abstract

Modern retrieval systems often struggle with upgrading to new and more powerful models due to the incompatibility of embeddings between the old and new models. This necessitates a costly process known as backfilling, which involves re-computing the embeddings for a large number of data samples. In vision, Backward-compatible Training (BT) has been proposed to ensure that the new model aligns with the old model's embeddings. This paper extends the concept of vision-only BT to the field of cross-modal retrieval, marking the first attempt to address Cross-modal BT (XBT). Our goal is to achieve backward-compatibility between Vision-Language Pretraining (VLP) models, such as CLIP, for the cross-modal retrieval task. To address XBT challenges, we propose an efficient solution: a projection module that maps the new model's embeddings to those of the old model. This module, pretrained solely with text data, significantly reduces the number of image-text pairs required for XBT learning, and, once it is pretrained, it avoids using the old model during training. Furthermore, we utilize parameter-efficient training strategies that improve efficiency and preserve the off-the-shelf new model's knowledge by avoiding any modifications. Experimental results on cross-modal retrieval datasets demonstrate the effectiveness of XBT and its potential to enable backfill-free upgrades when a new VLP model emerges.