Neural Networks with Sparse Activation Induced by Large Bias: Tighter Analysis with Bias-Generalized NTK

TL;DR

This paper analyzes one-hidden-layer ReLU networks with large bias initialization in the NTK regime, revealing sparse activation, a new bias-generalized NTK, and improved convergence and generalization bounds.

Contribution

It introduces the bias-generalized NTK, characterizes its properties, and demonstrates faster convergence and sparsity-dependent generalization bounds for networks with large biases.

Findings

Networks with large bias have sparse activation during training.

The bias-generalized NTK differs from the standard NTK and has favorable properties.

Convergence speed is comparable to dense networks, with improved width requirements.

Abstract

We study training one-hidden-layer ReLU networks in the neural tangent kernel (NTK) regime, where the networks' biases are initialized to some constant rather than zero. We prove that under such initialization, the neural network will have sparse activation throughout the entire training process, which enables fast training procedures via some sophisticated computational methods. With such initialization, we show that the neural networks possess a different limiting kernel which we call \textit{bias-generalized NTK}, and we study various properties of the neural networks with this new kernel. We first characterize the gradient descent dynamics. In particular, we show that the network in this case can achieve as fast convergence as the dense network, as opposed to the previous work suggesting that the sparse networks converge slower. In addition, our result improves the previous required width to ensure convergence. Secondly, we study the networks' generalization: we show a width-sparsity dependence, which yields a sparsity-dependent Rademacher complexity and generalization bound. To our knowledge, this is the first sparsity-dependent generalization result via Rademacher complexity. Lastly, we study the smallest eigenvalue of this new kernel. We identify a data-dependent region where we can derive a much sharper lower bound on the NTK's smallest eigenvalue than the worst-case bound previously known. This can lead to improvement in the generalization bound.