TokenBlowUp: Resolving Representational Singularities in LLM Token Spaces via Monoidal Transformations

TL;DR

This paper introduces a novel scheme-theoretic approach to resolve geometric singularities in LLM token embeddings, leading to more stable and semantically disambiguated representations, and suggests a shift towards dynamic geometric computation in language models.

Contribution

It formalizes the problem of singularities in token spaces using scheme theory and proposes a blow-up method to achieve representational desingularization, a novel approach in this context.

Findings

Proves a theorem guaranteeing geometric regularization after blow-up.

Identifies the exceptional divisor as a space of semantic disambiguation.

Outlines architectural implications for dynamic geometric computation.

Abstract

Recent work has provided compelling evidence challenging the foundational manifold hypothesis for the token embedding spaces of Large Language Models (LLMs). These findings reveal the presence of geometric singularities around polysemous tokens, which can lead to representational instability. Existing methodologies, which presuppose a smooth data manifold, are ill-equipped to address such intrinsic structural flaws. In this paper, we formalize this problem in the language of scheme theory and propose a rigorous resolution by applying the scheme-theoretic blow-up at each singular point. This procedure replaces a singular point in the ambient affine scheme with its exceptional divisor, which we identify as a canonical geometric space -- a projective space of directions -- that houses the disambiguated semantic meanings of the token. This process of ``representational desingularization'' constructs a new geometric landscape for embeddings. We prove a formal theorem guaranteeing the geometric regularization of this new space, showing that the original pathologies are resolved. Finally, we outline the architectural implications of our framework, arguing for a paradigm shift from static look-ups to dynamic, geometrically-grounded computation.