Complex fractal trainability boundary can arise from trivial non-convexity

TL;DR

This paper demonstrates that simple non-convex perturbations to quadratic functions can create fractal boundaries in neural network trainability, influenced by perturbation roughness and other factors, affecting hyperparameter tuning.

Contribution

It reveals that trivial non-convex modifications can produce fractal trainability boundaries, providing insights into the loss landscape's complexity during neural network training.

Findings

Fractal boundaries can emerge from simple cosine perturbations.

Roughness of perturbation controls fractal dimension of trainability boundaries.

Transition from non-fractal to fractal boundaries occurs as roughness increases.

Abstract

Training neural networks involves optimizing parameters to minimize a loss function, where the nature of the loss function and the optimization strategy are crucial for effective training. Hyperparameter choices, such as the learning rate in gradient descent (GD), significantly affect the success and speed of convergence. Recent studies indicate that the boundary between bounded and divergent hyperparameters can be fractal, complicating reliable hyperparameter selection. However, the nature of this fractal boundary and methods to avoid it remain unclear. In this study, we focus on GD to investigate the loss landscape properties that might lead to fractal trainability boundaries. We discovered that fractal boundaries can emerge from simple non-convex perturbations, i.e., adding or multiplying cosine type perturbations to quadratic functions. The observed fractal dimensions are influenced by factors like parameter dimension, type of non-convexity, perturbation wavelength, and perturbation amplitude. Our analysis identifies "roughness of perturbation", which measures the gradient's sensitivity to parameter changes, as the factor controlling fractal dimensions of trainability boundaries. We observed a clear transition from non-fractal to fractal trainability boundaries as roughness increases, with the critical roughness causing the perturbed loss function non-convex. Thus, we conclude that fractal trainability boundaries can arise from very simple non-convexity. We anticipate that our findings will enhance the understanding of complex behaviors during neural network training, leading to more consistent and predictable training strategies.