Gradient representations in ReLU networks as similarity functions

TL;DR

This paper explores how the tangent space of ReLU networks can be used to define similarity functions, introducing a Riemannian metric that enhances decision refinement and similarity measures.

Contribution

It introduces a Riemannian metric on network parameters that acts as an effective similarity function, improving upon the original network's decision boundaries.

Findings

The Riemannian metric forms a similarity function comparable to the network's original output.

A sparse metric increases the similarity gap, enhancing decision refinement.

The approach provides a geometric perspective on network decision boundaries.

Abstract

Feed-forward networks can be interpreted as mappings with linear decision surfaces at the level of the last layer. We investigate how the tangent space of the network can be exploited to refine the decision in case of ReLU (Rectified Linear Unit) activations. We show that a simple Riemannian metric parametrized on the parameters of the network forms a similarity function at least as good as the original network and we suggest a sparse metric to increase the similarity gap.