Native Logical and Hierarchical Representations with Subspace Embeddings

TL;DR

This paper introduces a new embedding paradigm representing concepts as linear subspaces, enabling hierarchical, logical, and set-theoretic operations, and achieves state-of-the-art results in word and language inference tasks.

Contribution

It proposes a novel subspace embedding framework that models hierarchy and logic naturally, with a differentiable learning method for orientation and dimensionality.

Findings

State-of-the-art in WordNet reconstruction and link prediction

Outperforms bi-encoder baselines on natural language inference

Provides an interpretable, geometrically grounded logical entailment model

Abstract

Traditional neural embeddings represent concepts as points, excelling at similarity but struggling with higher-level reasoning and asymmetric relationships. We introduce a novel paradigm: embedding concepts as linear subspaces. This framework inherently models generality via subspace dimensionality and hierarchy through subspace inclusion. It naturally supports set-theoretic operations like intersection (conjunction), linear sum (disjunction) and orthogonal complements (negations), aligning with classical formal semantics. To enable differentiable learning, we propose a smooth relaxation of orthogonal projection operators, allowing for the learning of both subspace orientation and dimension. Our method achieves state-of-the-art results in reconstruction and link prediction on WordNet. Furthermore, on natural language inference benchmarks, our subspace embeddings surpass bi-encoder baselines, offering an interpretable formulation of entailment that is both geometrically grounded and amenable to logical operations.