Rethinking Benchmarks for Differentially Private Image Classification

TL;DR

This paper proposes a comprehensive benchmarking framework for differentially private image classification, enabling evaluation across diverse settings and fostering community progress through a public leaderboard.

Contribution

It introduces a new set of benchmarks for differentially private image classification and evaluates existing techniques across these settings.

Findings

Established techniques vary in effectiveness across benchmarks

Benchmark diversity reveals strengths and limitations of current methods

Public leaderboard encourages ongoing research and comparison

Abstract

We revisit benchmarks for differentially private image classification. We suggest a comprehensive set of benchmarks, allowing researchers to evaluate techniques for differentially private machine learning in a variety of settings, including with and without additional data, in convex settings, and on a variety of qualitatively different datasets. We further test established techniques on these benchmarks in order to see which ideas remain effective in different settings. Finally, we create a publicly available leader board for the community to track progress in differentially private machine learning.