Situating Sentence Embedders with Nearest Neighbor Overlap

TL;DR

This paper introduces nearest neighbor overlap (N2O), a simple, task-agnostic metric for comparing sentence embedders based on neighbor overlap, revealing how design choices influence embedder similarity.

Contribution

We propose N2O, a novel, straightforward method for comparing sentence embedders without relying on benchmark tasks or linguistic probes.

Findings

N2O effectively measures embedder similarity across different architectures.

Design choices significantly impact the similarity of sentence embedders.

N2O provides insights into embedder behavior beyond traditional benchmarks.

Abstract

As distributed approaches to natural language semantics have developed and diversified, embedders for linguistic units larger than words have come to play an increasingly important role. To date, such embedders have been evaluated using benchmark tasks (e.g., GLUE) and linguistic probes. We propose a comparative approach, nearest neighbor overlap (N2O), that quantifies similarity between embedders in a task-agnostic manner. N2O requires only a collection of examples and is simple to understand: two embedders are more similar if, for the same set of inputs, there is greater overlap between the inputs' nearest neighbors. Though applicable to embedders of texts of any size, we focus on sentence embedders and use N2O to show the effects of different design choices and architectures.