The Vanishing Decision Boundary Complexity and the Strong First Component

TL;DR

This paper reveals that deep models' decision boundaries are simple after training, with complex structures only appearing temporarily during training, and explores how predecessor models' boundaries relate to generalization.

Contribution

It demonstrates the vanishing complexity of decision boundaries in deep models and introduces methods to study generalization using predecessor models' boundaries.

Findings

Decision boundaries are simple post-training, complex structures vanish early.

Predecessor models' boundaries reflect the final model's generalization.

Insights into the first principal component, optimizer singularity, and skip connections effects.

Abstract

We show that unlike machine learning classifiers, there are no complex boundary structures in the decision boundaries for well-trained deep models. However, we found that the complicated structures do appear in training but they vanish shortly after shaping. This is a pessimistic news if one seeks to capture different levels of complexity in the decision boundary for understanding generalization, which works well in machine learning. Nonetheless, we found that the decision boundaries of predecessor models on the training data are reflective of the final model's generalization. We show how to use the predecessor decision boundaries for studying the generalization of deep models. We have three major findings. One is on the strength of the first principle component of deep models, another about the singularity of optimizers, and the other on the effects of the skip connections in ResNets. Code is at https://github.com/hengshu1/decision_boundary_github.