When Annotators Disagree, Topology Explains: Mapper, a Topological Tool for Exploring Text Embedding Geometry and Ambiguity

TL;DR

This paper introduces Mapper, a topological data analysis tool, to explore how language models encode ambiguity and structure in embedding space, revealing insights beyond traditional visualization methods.

Contribution

It demonstrates that Mapper uncovers the topological structure of model embeddings, highlighting decision regions and ambiguity in a way that traditional methods cannot.

Findings

Mapper reveals modular, non-convex regions aligned with model predictions.

Over 98% of connected components have high prediction purity.

Alignment with ground-truth labels decreases in ambiguous cases.

Abstract

Language models are often evaluated with scalar metrics like accuracy, but such measures fail to capture how models internally represent ambiguity, especially when human annotators disagree. We propose a topological perspective to analyze how fine-tuned models encode ambiguity and more generally instances. Applied to RoBERTa-Large on the MD-Offense dataset, Mapper, a tool from topological data analysis, reveals that fine-tuning restructures embedding space into modular, non-convex regions aligned with model predictions, even for highly ambiguous cases. Over $98\%$ of connected components exhibit $\geq 90\%$ prediction purity, yet alignment with ground-truth labels drops in ambiguous data, surfacing a hidden tension between structural confidence and label uncertainty. Unlike traditional tools such as PCA or UMAP, Mapper captures this geometry directly uncovering decision regions, boundary collapses, and overconfident clusters. Our findings position Mapper as a powerful diagnostic tool for understanding how models resolve ambiguity. Beyond visualization, it also enables topological metrics that may inform proactive modeling strategies in subjective NLP tasks.