Computationally Efficient Replicable Learning of Parities

TL;DR

This paper introduces the first efficient replicable learning algorithm for parity functions over arbitrary distributions, bridging the gap between replicability and differential privacy, and surpassing the limitations of the SQ model.

Contribution

It presents the first computationally efficient replicable algorithm for learning parities over any distribution, extending beyond SQ-learnable tasks.

Findings

Efficient replicable learning of parities over arbitrary distributions is possible.

Replicable learning can be more powerful than SQ-learning in certain tasks.

The new algorithm outputs a subspace covering most vectors in polynomial time.

Abstract

We study the computational relationship between replicability (Impagliazzo et al. [STOC `22], Ghazi et al. [NeurIPS `21]) and other stability notions. Specifically, we focus on replicable PAC learning and its connections to differential privacy (Dwork et al. [TCC 2006]) and to the statistical query (SQ) model (Kearns [JACM `98]). Statistically, it was known that differentially private learning and replicable learning are equivalent and strictly more powerful than SQ-learning. Yet, computationally, all previously known efficient (i.e., polynomial-time) replicable learning algorithms were confined to SQ-learnable tasks or restricted distributions, in contrast to differentially private learning. Our main contribution is the first computationally efficient replicable algorithm for realizable learning of parities over arbitrary distributions, a task that is known to be hard in the SQ-model, but possible under differential privacy. This result provides the first evidence that efficient replicable learning over general distributions strictly extends efficient SQ-learning, and is closer in power to efficient differentially private learning, despite computational separations between replicability and privacy. Our main building block is a new, efficient, and replicable algorithm that, given a set of vectors, outputs a subspace of their linear span that covers most of them.