Differentially Private Multimodal In-Context Learning

TL;DR

This paper introduces DP-MTV, a novel framework for many-shot multimodal in-context learning that maintains formal differential privacy, enabling sensitive data applications with minimal performance loss.

Contribution

DP-MTV is the first method to achieve differentially private multimodal in-context learning with hundreds of demonstrations, using a compact task vector aggregation approach.

Findings

Achieves 50% accuracy on VizWiz at ε=1.0, close to 55% non-private.

Supports multiple VLM architectures and auxiliary data.

Requires only a single noise addition for unlimited inference.

Abstract

Vision-language models are increasingly applied to sensitive domains such as medical imaging and personal photographs, yet existing differentially private methods for in-context learning are limited to few-shot, text-only settings because privacy cost scales with the number of tokens processed. We present Differentially Private Multimodal Task Vectors (DP-MTV), the first framework enabling many-shot multimodal in-context learning with formal $(\varepsilon, \delta)$-differential privacy by aggregating hundreds of demonstrations into compact task vectors in activation space. DP-MTV partitions private data into disjoint chunks, applies per-layer clipping to bound sensitivity, and adds calibrated noise to the aggregate, requiring only a single noise addition that enables unlimited inference queries. We evaluate on eight benchmarks across three VLM architectures, supporting deployment with or without auxiliary data. At $\varepsilon=1.0$, DP-MTV achieves 50% on VizWiz compared to 55% non-private and 35% zero-shot, preserving most of the gain from in-context learning under meaningful privacy constraints.