Neural Networks Learn Distance Metrics

TL;DR

This paper investigates how neural networks naturally favor distance-based representations over intensity-based ones, affecting performance, and introduces a new architecture, OffsetL2, to validate this geometric framework.

Contribution

It demonstrates the impact of representation type on neural network performance and proposes a novel distance-based architecture, OffsetL2, grounded in a new geometric framework.

Findings

Distance representations improve model performance

OffsetL2 architecture validates the geometric framework

Distance-based learning is crucial in neural network design

Abstract

Neural networks may naturally favor distance-based representations, where smaller activations indicate closer proximity to learned prototypes. This contrasts with intensity-based approaches, which rely on activation magnitudes. To test this hypothesis, we conducted experiments with six MNIST architectural variants constrained to learn either distance or intensity representations. Our results reveal that the underlying representation affects model performance. We develop a novel geometric framework that explains these findings and introduce OffsetL2, a new architecture based on Mahalanobis distance equations, to further validate this framework. This work highlights the importance of considering distance-based learning in neural network design.