Noise-Tolerant Learning, the Parity Problem, and the Statistical Query Model

TL;DR

This paper presents a noise-tolerant learning algorithm for parity functions with certain dependencies, demonstrating that some problems not learnable in the statistical query model are learnable in the PAC model, and explores extensions of the statistical query model.

Contribution

It introduces the first efficient noise-tolerant algorithm for a class of parity functions and analyzes the limitations of extended statistical query models in learning.

Findings

Polynomial-time algorithm for decoding noisy linear codes with small k

Demonstrates strict subset relationship between learnable problems in SQ and PAC models

Extended t-wise query model does not increase the class of weakly learnable functions

Abstract

We describe a slightly sub-exponential time algorithm for learning parity functions in the presence of random classification noise. This results in a polynomial-time algorithm for the case of parity functions that depend on only the first O(log n log log n) bits of input. This is the first known instance of an efficient noise-tolerant algorithm for a concept class that is provably not learnable in the Statistical Query model of Kearns. Thus, we demonstrate that the set of problems learnable in the statistical query model is a strict subset of those problems learnable in the presence of noise in the PAC model. In coding-theory terms, what we give is a poly(n)-time algorithm for decoding linear k by n codes in the presence of random noise for the case of k = c log n loglog n for some c > 0. (The case of k = O(log n) is trivial since one can just individually check each of the 2^k possible messages and choose the one that yields the closest codeword.) A natural extension of the statistical query model is to allow queries about statistical properties that involve t-tuples of examples (as opposed to single examples). The second result of this paper is to show that any class of functions learnable (strongly or weakly) with t-wise queries for t = O(log n) is also weakly learnable with standard unary queries. Hence this natural extension to the statistical query model does not increase the set of weakly learnable functions.