Embedding principle of homogeneous neural network for classification problem

TL;DR

This paper introduces the KKT point embedding principle for homogeneous neural networks, showing how solutions in smaller networks can be embedded into larger ones and how this relates to training dynamics and network structure.

Contribution

It formalizes the KKT point embedding principle for homogeneous neural networks and connects static embeddings to gradient flow dynamics during training.

Findings

KKT points of smaller networks embed into larger networks via linear isometric transformations.

Training trajectories preserve the embedding, maintaining solution alignment.

Insights into network width effects and parameter redundancy in homogeneous networks.

Abstract

In this paper, we study the Karush-Kuhn-Tucker (KKT) points of the associated maximum-margin problem in homogeneous neural networks, including fully-connected and convolutional neural networks. In particular, We investigates the relationship between such KKT points across networks of different widths generated. We introduce and formalize the \textbf{KKT point embedding principle}, establishing that KKT points of a homogeneous network's max-margin problem ($P_{\Phi}$) can be embedded into the KKT points of a larger network's problem ($P_{\tilde{\Phi}}$) via specific linear isometric transformations. We rigorously prove this principle holds for neuron splitting in fully-connected networks and channel splitting in convolutional neural networks. Furthermore, we connect this static embedding to the dynamics of gradient flow training with smooth losses. We demonstrate that trajectories initiated from appropriately mapped points remain mapped throughout training and that the resulting $\omega$-limit sets of directions are correspondingly mapped, thereby preserving the alignment with KKT directions dynamically when directional convergence occurs. We conduct several experiments to justify that trajectories are preserved. Our findings offer insights into the effects of network width, parameter redundancy, and the structural connections between solutions found via optimization in homogeneous networks of varying sizes.