Why not to use Cosine Similarity between Label Representations

TL;DR

This paper demonstrates that cosine similarity between label representations in softmax classifiers does not reliably reflect model behavior or probabilities, challenging common interpretability practices.

Contribution

It proves that cosine similarity of label representations can be arbitrarily manipulated without changing model outputs, highlighting its unreliability for interpretation.

Findings

Cosine similarity does not correlate with model probabilities.

Equivalent models can have label representations with cosine similarity of 1 or -1.

Centering representations does not resolve the interpretability issue.

Abstract

Cosine similarity is often used to measure the similarity of vectors. These vectors might be the representations of neural network models. However, it is not guaranteed that cosine similarity of model representations will tell us anything about model behaviour. In this paper we show that when using a softmax classifier, be it an image classifier or an autoregressive language model, measuring the cosine similarity between label representations (called unembeddings in the paper) does not give any information on the probabilities assigned by the model. Specifically, we prove that for any softmax classifier model, given two label representations, it is possible to make another model which gives the same probabilities for all labels and inputs, but where the cosine similarity between the representations is now either 1 or -1. We give specific examples of models with very high or low cosine simlarity between representations and show how to we can make equivalent models where the cosine similarity is now -1 or 1. This translation ambiguity can be fixed by centering the label representations, however, labels with representations with low cosine similarity can still have high probability for the same inputs. Fixing the length of the representations still does not give a guarantee that high(or low) cosine similarity will give high(or low) probability to the labels for the same inputs. This means that when working with softmax classifiers, cosine similarity values between label representations should not be used to explain model probabilities.