Rethinking Word Similarity: Semantic Similarity through Classification Confusion

TL;DR

This paper introduces Word Confusion, a novel similarity measure based on classifier confusion that captures the dynamic, context-dependent, and asymmetrical nature of semantic similarity, outperforming traditional cosine methods in various tasks.

Contribution

The paper proposes Word Confusion, a classifier-based similarity measure that incorporates dynamic features and context, offering a more nuanced understanding of semantic similarity than existing methods.

Findings

Comparable to cosine similarity in matching human judgments

Able to incorporate predetermined features of interest

Effectively captures meaning change over time

Abstract

Word similarity has many applications to social science and cultural analytics tasks like measuring meaning change over time and making sense of contested terms. Yet traditional similarity methods based on cosine similarity between word embeddings cannot capture the context-dependent, asymmetrical, polysemous nature of semantic similarity. We propose a new measure of similarity, Word Confusion, that reframes semantic similarity in terms of feature-based classification confusion. Word Confusion is inspired by Tversky's suggestion that similarity features be chosen dynamically. Here we train a classifier to map contextual embeddings to word identities and use the classifier confusion (the probability of choosing a confounding word c instead of the correct target word t) as a measure of the similarity of c and t. The set of potential confounding words acts as the chosen features. Our method is comparable to cosine similarity in matching human similarity judgments across several datasets (MEN, WirdSim353, and SimLex), and can measure similarity using predetermined features of interest. We demonstrate our model's ability to make use of dynamic features by applying it to test a hypothesis about changes in the 18th C. meaning of the French word "revolution" from popular to state action during the French Revolution. We hope this reimagining of semantic similarity will inspire the development of new tools that better capture the multi-faceted and dynamic nature of language, advancing the fields of computational social science and cultural analytics and beyond.