A Metric Topology of Deep Learning for Data Classification

TL;DR

This paper introduces a novel metric topology for deep learning networks based on classification performance, providing a mathematical framework to better understand DL models beyond the traditional parameter space.

Contribution

It proposes a new probabilistic distance measure that groups networks with similar classification outcomes into equivalence classes, establishing a metric space structure for DL networks.

Findings

The proposed metric is provably a valid metric on the quotient set.

Under mild conditions, the metric space is compact and aligns with the quotient topological space.

The framework offers a new mathematical perspective for analyzing deep learning models.

Abstract

Empirically, Deep Learning (DL) has demonstrated unprecedented success in practical applications. However, DL remains by and large a mysterious "black-box", spurring recent theoretical research to build its mathematical foundations. In this paper, we investigate DL for data classification through the prism of metric topology. Considering that conventional Euclidean metric over the network parameter space typically fails to discriminate DL networks according to their classification outcomes, we propose from a probabilistic point of view a meaningful distance measure, whereby DL networks yielding similar classification performances are close. The proposed distance measure defines such an equivalent relation among network parameter vectors that networks performing equally well belong to the same equivalent class. Interestingly, our proposed distance measure can provably serve as a metric on the quotient set modulo the equivalent relation. Then, under quite mild conditions it is shown that, apart from a vanishingly small subset of networks likely to predict non-unique labels, our proposed metric space is compact, and coincides with the well-known quotient topological space. Our study contributes to fundamental understanding of DL, and opens up new ways of studying DL using fruitful metric space theory.