Limitation of Characterizing Implicit Regularization by Data-independent Functions

TL;DR

This paper investigates the limitations of characterizing neural network implicit regularization using data-independent functions, highlighting the profound data dependency and proposing mechanisms that demonstrate these limitations.

Contribution

It introduces two dynamical mechanisms showing that certain neural networks cannot be fully characterized by data-independent functions, emphasizing data dependency in implicit regularization.

Findings

Data-independent functions cannot fully characterize some neural networks.

Two-point and One-point Overlapping mechanisms demonstrate these limitations.

Results highlight the importance of data dependency in understanding implicit regularization.

Abstract

In recent years, understanding the implicit regularization of neural networks (NNs) has become a central task in deep learning theory. However, implicit regularization is itself not completely defined and well understood. In this work, we attempt to mathematically define and study implicit regularization. Importantly, we explore the limitations of a common approach to characterizing implicit regularization using data-independent functions. We propose two dynamical mechanisms, i.e., Two-point and One-point Overlapping mechanisms, based on which we provide two recipes for producing classes of one-hidden-neuron NNs that provably cannot be fully characterized by a type of or all data-independent functions. Following the previous works, our results further emphasize the profound data dependency of implicit regularization in general, inspiring us to study in detail the data dependency of NN implicit regularization in the future.