Learning a Single Neuron for Non-monotonic Activation Functions

TL;DR

This paper proves that single neurons with certain non-monotonic activation functions like SiLU, Swish, and GELU can be learned efficiently using gradient descent under Gaussian inputs, filling a gap in theoretical understanding.

Contribution

It establishes polynomial-time learnability of single neurons with non-monotonic activations, extending previous results limited to monotonic functions, under mild input distribution conditions.

Findings

Learnability guaranteed for non-monotonic activations like SiLU, Swish, GELU

Polynomial sample complexity and runtime under Gaussian inputs

Conditions on activation functions are satisfied by practical non-monotonic functions

Abstract

We study the problem of learning a single neuron $\mathbf{x}\mapsto \sigma(\mathbf{w}^T\mathbf{x})$ with gradient descent (GD). All the existing positive results are limited to the case where $\sigma$ is monotonic. However, it is recently observed that non-monotonic activation functions outperform the traditional monotonic ones in many applications. To fill this gap, we establish learnability without assuming monotonicity. Specifically, when the input distribution is the standard Gaussian, we show that mild conditions on $\sigma$ (e.g., $\sigma$ has a dominating linear part) are sufficient to guarantee the learnability in polynomial time and polynomial samples. Moreover, with a stronger assumption on the activation function, the condition of input distribution can be relaxed to a non-degeneracy of the marginal distribution. We remark that our conditions on $\sigma$ are satisfied by practical non-monotonic activation functions, such as SiLU/Swish and GELU. We also discuss how our positive results are related to existing negative results on training two-layer neural networks.