Gauge-invariant representation holonomy
Vasileios Sevetlidis, George Pavlidis
TL;DR
This paper introduces representation holonomy, a gauge-invariant measure of how learned features in deep networks change along input paths, revealing hidden curvature and improving understanding of model robustness and training dynamics.
Contribution
The paper proposes a novel gauge-invariant statistic called representation holonomy that captures path-dependent geometric properties of neural representations, addressing limitations of existing similarity measures.
Findings
Holonomy increases with loop radius, indicating more complex geometry.
Holonomy distinguishes models with similar CKA scores but different robustness.
Holonomy correlates with adversarial and corruption robustness, and tracks training dynamics.
Abstract
Deep networks learn internal representations whose geometry--how features bend, rotate, and evolve--affects both generalization and robustness. Existing similarity measures such as CKA or SVCCA capture pointwise overlap between activation sets, but miss how representations change along input paths. Two models may appear nearly identical under these metrics yet respond very differently to perturbations or adversarial stress. We introduce representation holonomy, a gauge-invariant statistic that measures this path dependence. Conceptually, holonomy quantifies the "twist" accumulated when features are parallel-transported around a small loop in input space: flat representations yield zero holonomy, while nonzero values reveal hidden curvature. Our estimator fixes gauge through global whitening, aligns neighborhoods using shared subspaces and rotation-only Procrustes, and embeds the result…
Peer Reviews
Decision·ICLR 2026 Poster
This is an important topic. ML networks are highly redundant in their parameterization and optimization landscapes, and it is critical for the field to develop measures that can characterize fundamental properties for use in analysis/comparison/design. The paper is built on a mathematical foundation that is rich, and perhaps not so familiar to a large portion of the NeurIPS community. The authors have generally done a good job of defining and explaining their construction, and the community w
The main weakness, for me, is that after several passes I am still uncertain of exactly why the properties captured by holonomy (distortion of geometry along closed-loop paths) are essential to understanding or comparing network functionality. Specifically, under what conditions are the properties captured by holonomy, but not captured by current alignment methods (CCA, RSA) or differential analalyses (Lie derivatives, or simply comparison of local Jacobians), essential for evaluating or improv
* Holonomy captures properties of a network that aren't measured by other representation similarity metrics. In particular, this metric captures the dynamics of representations as inputs change, which may make it more relevant to studies of properties like robustness than traditional RS metrics. * Holonomy can be used in conjunction with existing RS metrics to allow for a significantly deeper understanding of learned representations, as shown in section 5.3. * Estimates of error are included wit
* The theory is somewhat difficult to follow. A reader coming from the representation similarity literature might not have a background in the mathematics that are relevant to this paper. I think that defining terms more clearly and pointing readers to relevant background would be helpful. For example, gauge-invariance is never defined despite being in the title. Given the breadth of the ICLR audience, this type of knowledge cannot be assumed. * Similarly, the inclusion of proof sketches would b
* Conceptual originality: Introducing holonomy—a gauge-invariant geometric notion from differential geometry—into representation analysis is creative and nontrivial. * Mathematical rigor: The formal statements (gauge invariance, affine invariance, small-radius behavior) are internally consistent using matrix perturbation tools. Although I have to admit that I didn’t get a chance to go over all the appendices. * Potential significance: In principle, this idea could offer new insights for feature-
* Incomplete and inconclusive experiments: The reported results (e.g., Figs. 2–3) show overlapping confidence intervals, and several key correlations (Table 2) are weak or include zero within the 95% CI. Without stronger or broader evidence, the empirical support for holonomy as a meaningful metric remains limited. * Missing comparisons to existing metrics: Although the paper discusses CKA and CCA extensively, it never provides side-by-side empirical comparisons. Such baselines are crucial to de
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsGenerative Adversarial Networks and Image Synthesis · Stochastic Gradient Optimization Techniques · Advanced Graph Neural Networks