Differentially Private Representation Learning via Image Captioning

TL;DR

This paper introduces a novel method for differentially private representation learning using image captioning, achieving high-quality image features on large-scale datasets and significantly improving privacy-accuracy trade-offs in vision tasks.

Contribution

The authors develop a scalable DP image captioning model trained on large multimodal datasets, enabling high-quality private image representations for downstream tasks.

Findings

DP-Cap achieves 65.8% accuracy on ImageNet-1K under privacy budget ε=8.

Significant improvement over previous state-of-the-art (56.5%).

Effective training on 233M images from LAION-2B dataset.

Abstract

Differentially private (DP) machine learning is considered the gold-standard solution for training a model from sensitive data while still preserving privacy. However, a major barrier to achieving this ideal is its sub-optimal privacy-accuracy trade-off, which is particularly visible in DP representation learning. Specifically, it has been shown that under modest privacy budgets, most models learn representations that are not significantly better than hand-crafted features. In this work, we show that effective DP representation learning can be done via image captioning and scaling up to internet-scale multimodal datasets. Through a series of engineering tricks, we successfully train a DP image captioner (DP-Cap) on a 233M subset of LAION-2B from scratch using a reasonable amount of computation, and obtaining unprecedented high-quality image features that can be used in a variety of downstream vision and vision-language tasks. For example, under a privacy budget of $\varepsilon=8$ for the LAION dataset, a linear classifier trained on top of learned DP-Cap features attains $65.8\%$ accuracy on ImageNet-1K, considerably improving the previous SOTA of $56.5\%$.