Little Data, Big Impact: Privacy-Aware Visual Language Models via Minimal Tuning

TL;DR

This paper demonstrates that privacy-aware visual language models can be effectively developed using minimal data and carefully designed benchmarks, significantly improving privacy sensitivity without sacrificing overall performance.

Contribution

The authors introduce high-quality privacy benchmarks and a minimal instruction-tuning dataset, enabling substantial privacy-awareness improvements with only 100 samples.

Findings

Privacy benchmarks PrivBench and PrivBench-H are effective for evaluation.

Fine-tuning with 100 samples improves privacy sensitivity significantly.

Privacy-aware models outperform GPT-4 on privacy benchmarks.

Abstract

As Visual Language Models (VLMs) become increasingly embedded in everyday applications, ensuring they can recognize and appropriately handle privacy-sensitive content is essential. We conduct a comprehensive evaluation of ten state-of-the-art VLMs and identify limitations in their understanding of visual privacy. Existing datasets suffer from label inconsistencies, limiting their reliability. To address this, we introduce two compact, high-quality benchmarks, PrivBench and PrivBench-H, that focus on commonly recognized privacy categories aligned with the General Data Protection Regulation (GDPR). Additionally, we present PrivTune, an instruction-tuning dataset specifically curated to improve privacy sensitivity. We obtain a Privacy VLM by fine-tuning an off-the-shelf VLM on only 100 samples from PrivTune, which leads to substantial gains on all benchmarks, surpassing GPT-4, while maintaining strong performance on other tasks. Our findings show that privacy-awareness in VLMs can be substantially improved with minimal data and careful dataset design, setting the stage for safer, more privacy-aligned AI systems.