Learning High-Degree Parities: The Crucial Role of the Initialization

TL;DR

This paper investigates how the initial weight distribution affects the ability of gradient descent to learn high-degree parities, revealing that certain initializations enable learning of almost-full parities while others hinder it.

Contribution

It demonstrates that the learnability of high-degree parities depends critically on the initial weight distribution, with specific initializations enabling or preventing learning.

Findings

Discrete Rademacher initialization enables learning of almost-full parities.

Gaussian perturbation with large standard deviation prevents learning.

Learnability threshold depends on the standard deviation of initial weights.

Abstract

Parities have become a standard benchmark for evaluating learning algorithms. Recent works show that regular neural networks trained by gradient descent can efficiently learn degree $k$ parities on uniform inputs for constant $k$, but fail to do so when $k$ and $d-k$ grow with $d$ (here $d$ is the ambient dimension). However, the case where $k=d-O_d(1)$ (almost-full parities), including the degree $d$ parity (the full parity), has remained unsettled. This paper shows that for gradient descent on regular neural networks, learnability depends on the initial weight distribution. On one hand, the discrete Rademacher initialization enables efficient learning of almost-full parities, while on the other hand, its Gaussian perturbation with large enough constant standard deviation $\sigma$ prevents it. The positive result for almost-full parities is shown to hold up to $\sigma=O(d^{-1})$, pointing to questions about a sharper threshold phenomenon. Unlike statistical query (SQ) learning, where a singleton function class like the full parity is trivially learnable, our negative result applies to a fixed function and relies on an initial gradient alignment measure of potential broader relevance to neural networks learning.