Neural Networks Use Distance Metrics

TL;DR

This paper provides empirical evidence that neural networks with ReLU and Absolute Value activations learn distance-based representations, challenging the common intensity-based interpretation and revealing their sensitivity to distance perturbations.

Contribution

The study demonstrates that neural networks encode distance-based features and are highly sensitive to distance perturbations, offering new insights into their internal representations.

Findings

Neural networks are highly sensitive to small distance-based perturbations.

Models maintain robust performance under large intensity-based perturbations.

Findings challenge the prevailing intensity-based interpretation of activations.

Abstract

We present empirical evidence that neural networks with ReLU and Absolute Value activations learn distance-based representations. We independently manipulate both distance and intensity properties of internal activations in trained models, finding that both architectures are highly sensitive to small distance-based perturbations while maintaining robust performance under large intensity-based perturbations. These findings challenge the prevailing intensity-based interpretation of neural network activations and offer new insights into their learning and decision-making processes.