Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Experts

TL;DR

LIMoE is a novel sparse mixture of experts model that enables effective multimodal learning with images and text, achieving state-of-the-art zero-shot image classification performance while addressing training stability and expert utilization challenges.

Contribution

The paper introduces LIMoE, a multimodal sparse MoE model trained with contrastive loss, featuring an entropy-based regularization to improve stability and expert balance, with significant performance gains.

Findings

LIMoE achieves 78.6% zero-shot ImageNet accuracy with comparable compute to CLIP-L/14.

Scaling LIMoE to larger models yields 84.1% accuracy, competitive with larger models.

Expert layers organically develop modality-specific specialization.

Abstract

Large sparsely-activated models have obtained excellent performance in multiple domains. However, such models are typically trained on a single modality at a time. We present the Language-Image MoE, LIMoE, a sparse mixture of experts model capable of multimodal learning. LIMoE accepts both images and text simultaneously, while being trained using a contrastive loss. MoEs are a natural fit for a multimodal backbone, since expert layers can learn an appropriate partitioning of modalities. However, new challenges arise; in particular, training stability and balanced expert utilization, for which we propose an entropy-based regularization scheme. Across multiple scales, we demonstrate remarkable performance improvement over dense models of equivalent computational cost. LIMoE-L/16 trained comparably to CLIP-L/14 achieves 78.6% zero-shot ImageNet accuracy (vs. 76.2%), and when further scaled to H/14 (with additional data) it achieves 84.1%, comparable to state-of-the-art methods which use larger custom per-modality backbones and pre-training schemes. We analyse the quantitative and qualitative behavior of LIMoE, and demonstrate phenomena such as differing treatment of the modalities and the organic emergence of modality-specific experts.