On Geometric Structure of Activation Spaces in Neural Networks

TL;DR

This paper explores the geometric properties of activation spaces in neural networks, introduces an approximation algorithm for convex hulls, and proposes a new classification method that outperforms traditional neural network classifiers and indicates overfitting.

Contribution

It presents a novel approximation algorithm for high-dimensional convex hulls and reveals geometric properties of activation spaces, leading to a new classification approach and overfitting indicator.

Findings

Activation spaces share four common geometric properties.

The new classification method outperforms neural networks in certain tasks.

The method can serve as an indicator of overfitting.

Abstract

In this paper, we investigate the geometric structure of activation spaces of fully connected layers in neural networks and then show applications of this study. We propose an efficient approximation algorithm to characterize the convex hull of massive points in high dimensional space. Based on this new algorithm, four common geometric properties shared by the activation spaces are concluded, which gives a rather clear description of the activation spaces. We then propose an alternative classification method grounding on the geometric structure description, which works better than neural networks alone. Surprisingly, this data classification method can be an indicator of overfitting in neural networks. We believe our work reveals several critical intrinsic properties of modern neural networks and further gives a new metric for evaluating them.