Active Data Curation Effectively Distills Large-Scale Multimodal Models

TL;DR

This paper introduces ACED, a simple active data curation method that enhances contrastive multimodal pretraining, outperforming traditional knowledge distillation and achieving state-of-the-art results with less inference cost.

Contribution

It proposes ACID, an online batch selection strategy, and combines it with standard KD to create ACED, a scalable framework that improves multimodal model performance and efficiency.

Findings

ACED outperforms strong KD baselines across various tasks.

ACED achieves state-of-the-art results on 27 zero-shot tasks.

ACED models are effective vision encoders for multimodal generative tasks.

Abstract

Knowledge distillation (KD) is the de facto standard for compressing large-scale models into smaller ones. Prior works have explored ever more complex KD strategies involving different objective functions, teacher-ensembles, and weight inheritance. In this work we explore an alternative, yet simple approach -- active data curation as effective distillation for contrastive multimodal pretraining. Our simple online batch selection method, ACID, outperforms strong KD baselines across various model-, data- and compute-configurations. Further, we find such an active data curation strategy to in fact be complementary to standard KD, and can be effectively combined to train highly performant inference-efficient models. Our simple and scalable pretraining framework, ACED, achieves state-of-the-art results across 27 zero-shot classification and retrieval tasks with upto 11% less inference FLOPs. We further demonstrate that our ACED models yield strong vision-encoders for training generative multimodal models in the LiT-Decoder setting, outperforming larger vision encoders for image-captioning and visual question-answering tasks.