On the Complexity of Learning Sparse Functions with Statistical and Gradient Queries

TL;DR

This paper analyzes the complexity of gradient-based algorithms in learning sparse functions, introducing Differentiable Learning Queries ($ extsf{DLQ}$) to model gradient queries and characterizing their query complexity across different loss functions.

Contribution

It introduces $ extsf{DLQ}$ as a new query model for gradient algorithms and provides a tight characterization of their complexity for learning sparse functions under various loss functions.

Findings

$ extsf{DLQ}$ matches $ extsf{CSQ}$ complexity for squared loss.

For $ extsf{L}_1$ loss, $ extsf{DLQ}$ has the same complexity as $ extsf{SQ}.

$ extsf{DLQ}$ captures the complexity of learning with gradient descent in neural networks.

Abstract

The goal of this paper is to investigate the complexity of gradient algorithms when learning sparse functions (juntas). We introduce a type of Statistical Queries ($\mathsf{SQ}$), which we call Differentiable Learning Queries ($\mathsf{DLQ}$), to model gradient queries on a specified loss with respect to an arbitrary model. We provide a tight characterization of the query complexity of $\mathsf{DLQ}$ for learning the support of a sparse function over generic product distributions. This complexity crucially depends on the loss function. For the squared loss, $\mathsf{DLQ}$ matches the complexity of Correlation Statistical Queries $(\mathsf{CSQ})$--potentially much worse than $\mathsf{SQ}$. But for other simple loss functions, including the $\ell_1$ loss, $\mathsf{DLQ}$ always achieves the same complexity as $\mathsf{SQ}$. We also provide evidence that $\mathsf{DLQ}$ can indeed capture learning with (stochastic) gradient descent by showing it correctly describes the complexity of learning with a two-layer neural network in the mean field regime and linear scaling.