DisCo-CLIP: A Distributed Contrastive Loss for Memory Efficient CLIP Training

TL;DR

DisCo-CLIP introduces a distributed, memory-efficient contrastive loss computation method for CLIP training, significantly reducing GPU memory usage and enabling large-batch training on fewer GPUs without loss of accuracy.

Contribution

It decomposes contrastive loss into intra-GPU and inter-GPU parts, reducing memory from O(B^2) to O(B^2/N) and enabling efficient large-batch CLIP training.

Findings

Reduces GPU memory consumption for contrastive loss

Enables training with larger batch sizes on fewer GPUs

Maintains mathematical equivalence to original contrastive loss

Abstract

We propose DisCo-CLIP, a distributed memory-efficient CLIP training approach, to reduce the memory consumption of contrastive loss when training contrastive learning models. Our approach decomposes the contrastive loss and its gradient computation into two parts, one to calculate the intra-GPU gradients and the other to compute the inter-GPU gradients. According to our decomposition, only the intra-GPU gradients are computed on the current GPU, while the inter-GPU gradients are collected via all_reduce from other GPUs instead of being repeatedly computed on every GPU. In this way, we can reduce the GPU memory consumption of contrastive loss computation from $\bigO(B^2)$ to $\bigO(\frac{B^2}{N})$, where $B$ and $N$ are the batch size and the number of GPUs used for training. Such a distributed solution is mathematically equivalent to the original non-distributed contrastive loss computation, without sacrificing any computation accuracy. It is particularly efficient for large-batch CLIP training. For instance, DisCo-CLIP can enable contrastive training of a ViT-B/32 model with a batch size of 32K or 196K using 8 or 64 A100 40GB GPUs, compared with the original CLIP solution which requires 128 A100 40GB GPUs to train a ViT-B/32 model with a batch size of 32K. The code will be released at https://github.com/IDEA-Research/DisCo-CLIP