Differentially Private Contrastive Learning via Bounding Group-level Contribution

TL;DR

This paper introduces DP-GCL, a novel differentially private contrastive learning framework that reduces inter-sample dependency by bounding group-level contributions, leading to improved utility in private representation learning.

Contribution

It proposes a new DP contrastive learning method that limits gradient dependency through batch partitioning and intra-group augmentation, enhancing privacy-utility trade-offs.

Findings

DP-GCL improves image classification accuracy by 5.6%.

DP-GCL enhances image-text retrieval accuracy by 20.1%.

The method outperforms existing DP contrastive approaches across eight datasets.

Abstract

Differentially private (DP) contrastive learning aims to learn general-purpose representations from sensitive data, alleviating the privacy leakage concerns of organizations deploying or sharing embedding models trained on private user content. However, existing approaches suffer from severe utility degradation due to the over-strong inter-sample dependency inherent in standard contrastive objectives, where each sample's gradient depends on all other samples in the batch, amplifying the impact of DP noise. In this work, we argue that effective DP contrastive learning requires explicitly reducing such intrinsic inter-sample reliance. To this end, we propose DP-GCL, a principled DP contrastive learning framework that structurally limits gradient dependency through bounding group-level contribution. DP-GCL partitions each batch into small, disjoint groups and restricts available negative samples to within-group samples, thereby localizing gradient influence and reducing sensitivity. To counteract the resulting loss of negative sample diversity, we further introduce intra-group augmentation, which generates additional negative views without increasing privacy cost. Extensive experiments across eight datasets demonstrate that DP-GCL consistently advances the state of the art in both uni-modal and multi-modal contrastive learning under practical privacy budgets: it improves image classification accuracy by 5.6% and image-text retrieval accuracy by 20.1% over existing DP contrastive methods.