Taken by Surprise: Contrast effect for Similarity Scores

TL;DR

This paper introduces the surprise score, an ensemble-normalized similarity metric inspired by human perception, which improves zero- and few-shot classification and clustering performance over traditional cosine similarity.

Contribution

The paper proposes the surprise score, a novel ensemble-normalized similarity measure that captures contrast effects and enhances classification accuracy in NLP tasks.

Findings

10-15% improvement over cosine similarity in classification tasks

Significant performance gains in zero- and few-shot learning

Effective in clustering and similarity evaluation

Abstract

Accurately evaluating the similarity of object vector embeddings is of critical importance for natural language processing, information retrieval and classification tasks. Popular similarity scores (e.g cosine similarity) are based on pairs of embedding vectors and disregard the distribution of the ensemble from which objects are drawn. Human perception of object similarity significantly depends on the context in which the objects appear. In this work we propose the $\textit{surprise score}$, an ensemble-normalized similarity metric that encapsulates the contrast effect of human perception and significantly improves the classification performance on zero- and few-shot document classification tasks. This score quantifies the surprise to find a given similarity between two elements relative to the pairwise ensemble similarities. We evaluate this metric on zero/few shot classification and clustering tasks and typically find 10-15 % better performance compared to raw cosine similarity. Our code is available at https://github.com/MeetElise/surprise-similarity.