Unveiling Transformer Perception by Exploring Input Manifolds

TL;DR

This paper presents a mathematically grounded method to explore and interpret the input space of Transformer models by identifying equivalence classes and navigating between them, enhancing understanding of model perception.

Contribution

It introduces a novel approach based on eigendecomposition of the Jacobian to identify and explore equivalence classes in Transformer input space, enabling better interpretability.

Findings

Successfully reconstructs equivalence classes in input space

Enables navigation across different equivalence classes

Provides meaningful interpretation of retrieved instances

Abstract

This paper introduces a general method for the exploration of equivalence classes in the input space of Transformer models. The proposed approach is based on sound mathematical theory which describes the internal layers of a Transformer architecture as sequential deformations of the input manifold. Using eigendecomposition of the pullback of the distance metric defined on the output space through the Jacobian of the model, we are able to reconstruct equivalence classes in the input space and navigate across them. Our method enables two complementary exploration procedures: the first retrieves input instances that produce the same class probability distribution as the original instance-thus identifying elements within the same equivalence class-while the second discovers instances that yield a different class probability distribution, effectively navigating toward distinct equivalence classes. Finally, we demonstrate how the retrieved instances can be meaningfully interpreted by projecting their embeddings back into a human-readable format.