On Traceability in $\ell_p$ Stochastic Convex Optimization

TL;DR

This paper explores the relationship between traceability and excess risk in stochastic convex optimization under different $ extit{l}_p$ geometries, revealing phase transitions and lower bounds for private learning.

Contribution

It establishes fundamental tradeoffs between traceability and excess risk in SCO, including phase transition points and new lower bounds for differential privacy.

Findings

Traceability is linked to excess risk thresholds in SCO.

Phase transition occurs at certain excess risk levels for $p eq 2$.

New lower bounds for private learning in $ extit{l}_p$ geometries.

Abstract

In this paper, we investigate the necessity of traceability for accurate learning in stochastic convex optimization (SCO) under $\ell_p$ geometries. Informally, we say a learning algorithm is $m$-traceable if, by analyzing its output, it is possible to identify at least $m$ of its training samples. Our main results uncover a fundamental tradeoff between traceability and excess risk in SCO. For every $p\in [1,\infty)$, we establish the existence of an excess risk threshold below which every sample-efficient learner is traceable with the number of samples which is a constant fraction of its training sample. For $p\in [1,2]$, this threshold coincides with the best excess risk of differentially private (DP) algorithms, i.e., above this threshold, there exist algorithms that are not traceable, which corresponds to a sharp phase transition. For $p \in (2,\infty)$, this threshold instead gives novel lower bounds for DP learning, partially closing an open problem in this setup. En route to establishing these results, we prove a sparse variant of the fingerprinting lemma, which is of independent interest to the community.