All Roads Lead to Rome? Exploring the Invariance of Transformers' Representations

TL;DR

This paper investigates whether transformer models learn invariant representations across different training runs, proposing a bijective alignment method using invertible neural networks to analyze and interpret these representations.

Contribution

It introduces the Bijection Hypothesis and BERT-INN, a novel invertible neural network approach for aligning transformer representations, advancing interpretability research.

Findings

BERT-INN outperforms CCA in aligning representations

Transformers learn largely isomorphic representation spaces

Aligned embeddings reveal meaningful interpretability insights

Abstract

Transformer models bring propelling advances in various NLP tasks, thus inducing lots of interpretability research on the learned representations of the models. However, we raise a fundamental question regarding the reliability of the representations. Specifically, we investigate whether transformers learn essentially isomorphic representation spaces, or those that are sensitive to the random seeds in their pretraining process. In this work, we formulate the Bijection Hypothesis, which suggests the use of bijective methods to align different models' representation spaces. We propose a model based on invertible neural networks, BERT-INN, to learn the bijection more effectively than other existing bijective methods such as the canonical correlation analysis (CCA). We show the advantage of BERT-INN both theoretically and through extensive experiments, and apply it to align the reproduced BERT embeddings to draw insights that are meaningful to the interpretability research. Our code is at https://github.com/twinkle0331/BERT-similarity.