Semantic projection: recovering human knowledge of multiple, distinct object features from word embeddings

TL;DR

This paper introduces semantic projection, a method to extract human-like, context-dependent semantic features from word embeddings, demonstrating that these models encode rich, manipulable knowledge about object properties.

Contribution

The authors propose a novel semantic projection technique that reveals multiple, distinct object features from word embeddings, addressing the limitation of their perceived semantic rigidity.

Findings

Semantic projection recovers human judgments of object features.

Word embeddings encode diverse, context-dependent semantic knowledge.

Method demonstrates flexible manipulation of semantic features in vector space.

Abstract

The words of a language reflect the structure of the human mind, allowing us to transmit thoughts between individuals. However, language can represent only a subset of our rich and detailed cognitive architecture. Here, we ask what kinds of common knowledge (semantic memory) are captured by word meanings (lexical semantics). We examine a prominent computational model that represents words as vectors in a multidimensional space, such that proximity between word-vectors approximates semantic relatedness. Because related words appear in similar contexts, such spaces - called "word embeddings" - can be learned from patterns of lexical co-occurrences in natural language. Despite their popularity, a fundamental concern about word embeddings is that they appear to be semantically "rigid": inter-word proximity captures only overall similarity, yet human judgments about object similarities are highly context-dependent and involve multiple, distinct semantic features. For example, dolphins and alligators appear similar in size, but differ in intelligence and aggressiveness. Could such context-dependent relationships be recovered from word embeddings? To address this issue, we introduce a powerful, domain-general solution: "semantic projection" of word-vectors onto lines that represent various object features, like size (the line extending from the word "small" to "big"), intelligence (from "dumb" to "smart"), or danger (from "safe" to "dangerous"). This method, which is intuitively analogous to placing objects "on a mental scale" between two extremes, recovers human judgments across a range of object categories and properties. We thus show that word embeddings inherit a wealth of common knowledge from word co-occurrence statistics and can be flexibly manipulated to express context-dependent meanings.