TiC-CLIP: Continual Training of CLIP Models

TL;DR

This paper introduces TiC benchmarks for continual training of vision-language models, evaluates the temporal robustness of existing models, and proposes an efficient rehearsal-based training method that reduces computational costs.

Contribution

It presents the first large-scale web data benchmarks for time-continuous training of vision-language models and demonstrates an effective rehearsal-based training approach.

Findings

Existing models lose accuracy over time without updates.

Rehearsal-based continual training reduces compute by 2.5x.

New benchmarks enable evaluation of temporal robustness.

Abstract

Keeping large foundation models up to date on latest data is inherently expensive. To avoid the prohibitive costs of constantly retraining, it is imperative to continually train these models. This problem is exacerbated by the lack of any large scale continual learning benchmarks or baselines. We introduce the first set of web-scale Time-Continual (TiC) benchmarks for training vision-language models: TiC-DataComp, TiC-YFCC, and TiC-Redcaps. TiC-DataComp, our largest dataset, contains over 12.7B timestamped image-text pairs spanning 9 years (2014-2022). We first use our benchmarks to curate various dynamic evaluations to measure temporal robustness of existing models. We show OpenAI's CLIP (trained on data up to 2020) loses $\approx 8\%$ zero-shot accuracy on our curated retrieval task from 2021-2022 compared with more recently trained models in OpenCLIP repository. We then study how to efficiently train models on time-continuous data. We demonstrate that a simple rehearsal-based approach that continues training from the last checkpoint and replays old data reduces compute by $2.5\times$ when compared to the standard practice of retraining from scratch. Code is available at https://github.com/apple/ml-tic-clip.