Interplay between depth of neural networks and locality of target functions

TL;DR

This paper explores how the depth of neural networks interacts with the locality of target functions, revealing that depth benefits local functions but hinders global ones, highlighting limitations of existing theories.

Contribution

It introduces the concepts of $k$-local and $k$-global functions and analyzes how network depth affects learning these functions, addressing gaps in current theoretical understanding.

Findings

Depth improves learning of local functions.

Depth hampers learning of global functions.

Neural tangent kernel fails to capture this interplay.

Abstract

It has been recognized that heavily overparameterized deep neural networks (DNNs) exhibit surprisingly good generalization performance in various machine-learning tasks. Although benefits of depth have been investigated from different perspectives such as the approximation theory and the statistical learning theory, existing theories do not adequately explain the empirical success of overparameterized DNNs. In this work, we report a remarkable interplay between depth and locality of a target function. We introduce $k$-local and $k$-global functions, and find that depth is beneficial for learning local functions but detrimental to learning global functions. This interplay is not properly captured by the neural tangent kernel, which describes an infinitely wide neural network within the lazy learning regime.