Shrink the longest: improving latent space isotropy with symplicial geometry

TL;DR

This paper introduces a novel regularization method based on simplicial geometry to enhance latent space isotropy in transformer models, improving downstream performance without additional inference costs.

Contribution

The paper proposes a new regularization technique using simplicial geometry and persistent entropy to increase isotropy in embeddings, avoiding reparametrization and extra inference overhead.

Findings

Increased downstream task performance after applying the method

Significant reduction in latent space anisotropy during fine-tuning

Effective use of geometric structures without additional model reparametrization

Abstract

Although transformer-based models have been dominating the field of deep learning, various studies of their embedding space have shown that they suffer from "representation degeneration problem": embeddings tend to be distributed in a narrow cone, making the latent space highly anisotropic. Increasing the isotropy has shown to improve performance in downstream tasks both in static and contextual language models. However, most of approaches either add inference overhead or require substantial amount of data for model reparametrization. We propose a novel regularization technique based on simplicial geometry to improve the isotropy of latent representations. The core idea of our method is based on maximizing the persistent entropy of barcodes obtained using Vietoris-Rips filtration from contextual embeddings in the underlying latent space. We demonstrate that the method leads to an increase in downstream performance while significantly lowering the anisotropy during fine-tuning by exploiting existing geometric structures instead of reparametrization.