Understanding the geometry of deep learning with decision boundary volume

TL;DR

This paper introduces a geometric measure called decision boundary volume to analyze neural network performance, showing that smaller boundary volumes generally correlate with better accuracy and generalization, especially in convolutional architectures.

Contribution

The paper proposes a novel, theoretically grounded method to quantify decision boundary complexity via local surface volumes, applicable to high-dimensional deep learning models.

Findings

Decision boundary volume inversely correlates with classification accuracy in convolutional networks.

Smoother decision boundaries tend to improve performance for data-structured architectures.

Relationship between boundary volume and generalization varies with architecture type.

Abstract

For classification tasks, the performance of a deep neural network is determined by the structure of its decision boundary, whose geometry directly affects essential properties of the model, including accuracy and robustness. Motivated by a classical tube formula due to Weyl, we introduce a method to measure the decision boundary of a neural network through local surface volumes, providing a theoretically justifiable and efficient measure enabling a geometric interpretation of the effectiveness of the model applicable to the high dimensional feature spaces considered in deep learning. A smaller surface volume is expected to correspond to lower model complexity and better generalisation. We verify, on a number of image processing tasks with convolutional architectures that decision boundary volume is inversely proportional to classification accuracy. Meanwhile, the relationship between local surface volume and generalisation for fully connected architecture is observed to be less stable between tasks. Therefore, for network architectures suited to a particular data structure, we demonstrate that smoother decision boundaries lead to better performance, as our intuition would suggest.