Relative Representations: Topological and Geometric Perspectives

TL;DR

This paper enhances relative representations for zero-shot model stitching by introducing normalization for invariance and topological densification for better clustering, leading to improved performance.

Contribution

It proposes two novel improvements—normalization and topological densification—to enhance the robustness and effectiveness of relative representations.

Findings

Normalization improves invariance to rescaling and permutations.

Topological densification encourages class clustering during fine-tuning.

Enhanced methods lead to better zero-shot model stitching performance.

Abstract

Relative representations are an established approach to zero-shot model stitching, consisting of a non-trainable transformation of the latent space of a deep neural network. Based on insights of topological and geometric nature, we propose two improvements to relative representations. First, we introduce a normalization procedure in the relative transformation, resulting in invariance to non-isotropic rescalings and permutations. The latter coincides with the symmetries in parameter space induced by common activation functions. Second, we propose to deploy topological densification when fine-tuning relative representations, a topological regularization loss encouraging clustering within classes. We provide an empirical investigation on a natural language task, where both the proposed variations yield improved performance on zero-shot model stitching.