Diagnosing Neural Convergence with Topological Alignment Spectra

TL;DR

The paper introduces the Topological Alignment Spectrum (TAS), a multi-scale diagnostic tool that reveals detailed geometric and semantic alignment structures in neural networks, surpassing traditional scalar metrics.

Contribution

TAS provides a normalized, multi-scale measure of neural representational alignment, capturing local and global geometric relationships that are invisible to existing scalar metrics.

Findings

TAS distinguishes local jitter effects from macro-scale semantic reorganization.

Fine-tuning causes widespread topological reorganization across scales.

Semantic clusters dominate the alignment landscape across different models.

Abstract

Representational similarity in neural networks is inherently scale-dependent, yet widely used metrics such as Centered Kernel Alignment (CKA) and Procrustes analysis provide only global scalar estimates. These scalars often fail to distinguish micro-scale geometric jitter (local noise) from macro-scale semantic reorganization, compressing multi-scale structural relationships into a single uninformative value. We introduce the Topological Alignment Spectrum (TAS), a multi-scale diagnostic tool that sweeps normalized mean Jaccard similarity over varying neighborhood sizes. By normalizing the metric over an analytically-derived expected range (from expected overlap under randomness to perfect alignment), TAS yields a dimension-invariant metric over a spectrum of scales, where one indicates perfect structural alignment, zero reflects chance-level agreement, and negative values signal active anti-alignment at specific scales. Experiments on synthetic point clouds demonstrate that TAS allows the recognition of distinct types of alignment perturbation: local jitter harms fine-grained neighborhoods but preserves cluster-level structure, while cluster-center shuffling preserves local similarity but disrupts global alignment -- phenomena that remain invisible or conflated under global, single-scalar metrics. Applying TAS to the MultiBERTs collection reveals that fine-tuning induces comprehensive topological reorganization across scales, challenging the view of task adaptation as merely conservative or localized. While models from different random seeds remain locally divergent, semantic clusters emerge as the dominant scale of alignment. TAS thus offers a granular, topology-aware alternative for diagnosing convergence and representational stability in deep networks.