Recent Advances in Large Margin Learning

TL;DR

This survey reviews recent progress in large margin training for deep neural networks, discussing theoretical foundations, margin enlargement techniques, and their implications for generalization and robustness.

Contribution

It generalizes margin formulations to DNNs, categorizes recent methods, and explores theoretical links between margins, generalization, and robustness.

Findings

Categorized margin enlargement methods for DNNs.

Connected margins to network generalization and robustness.

Provided directions for future research in large margin DNN training.

Abstract

This paper serves as a survey of recent advances in large margin training and its theoretical foundations, mostly for (nonlinear) deep neural networks (DNNs) that are probably the most prominent machine learning models for large-scale data in the community over the past decade. We generalize the formulation of classification margins from classical research to latest DNNs, summarize theoretical connections between the margin, network generalization, and robustness, and introduce recent efforts in enlarging the margins for DNNs comprehensively. Since the viewpoint of different methods is discrepant, we categorize them into groups for ease of comparison and discussion in the paper. Hopefully, our discussions and overview inspire new research work in the community that aim to improve the performance of DNNs, and we also point to directions where the large margin principle can be verified to provide theoretical evidence why certain regularizations for DNNs function well in practice. We managed to shorten the paper such that the crucial spirit of large margin learning and related methods are better emphasized.