Cross-Model Semantics in Representation Learning

TL;DR

This paper explores how structural constraints in deep networks influence the stability and transferability of internal representations across different architectures, highlighting the role of inductive biases in improving model interoperability.

Contribution

It introduces a framework for analyzing representational alignment across diverse architectures using structural regularities and demonstrates their impact on transferability.

Findings

Structural regularities lead to more stable representational geometry.

Inductive biases enhance cross-model feature compatibility.

Regularized models show improved transferability across architectures.

Abstract

The internal representations learned by deep networks are often sensitive to architecture-specific choices, raising questions about the stability, alignment, and transferability of learned structure across models. In this paper, we investigate how structural constraints--such as linear shaping operators and corrective paths--affect the compatibility of internal representations across different architectures. Building on the insights from prior studies on structured transformations and convergence, we develop a framework for measuring and analyzing representational alignment across networks with distinct but related architectural priors. Through a combination of theoretical insights, empirical probes, and controlled transfer experiments, we demonstrate that structural regularities induce representational geometry that is more stable under architectural variation. This suggests that certain forms of inductive bias not only support generalization within a model, but also improve the interoperability of learned features across models. We conclude with a discussion on the implications of representational transferability for model distillation, modular learning, and the principled design of robust learning systems.