Lexicographic and Depth-Sensitive Margins in Homogeneous and Non-Homogeneous Deep Models

TL;DR

This paper investigates how infinitesimal regularization and gradient descent lead to margin-maximizing solutions in deep models, revealing differences between homogeneous and non-homogeneous architectures and their convergence properties.

Contribution

It extends previous work by analyzing both homogeneous and non-homogeneous deep models, characterizing their margin-maximizing solutions and the conditions for convergence under gradient descent.

Findings

Non-homogeneous models discard unnecessary shallow sub-models.

Homogeneous models converge to lexicographic max-margin solutions.

Conditions identified for max-margin solutions via gradient descent.

Abstract

With an eye toward understanding complexity control in deep learning, we study how infinitesimal regularization or gradient descent optimization lead to margin maximizing solutions in both homogeneous and non-homogeneous models, extending previous work that focused on infinitesimal regularization only in homogeneous models. To this end we study the limit of loss minimization with a diverging norm constraint (the "constrained path"), relate it to the limit of a "margin path" and characterize the resulting solution. For non-homogeneous ensemble models, which output is a sum of homogeneous sub-models, we show that this solution discards the shallowest sub-models if they are unnecessary. For homogeneous models, we show convergence to a "lexicographic max-margin solution", and provide conditions under which max-margin solutions are also attained as the limit of unconstrained gradient descent.