Embedding Style Beyond Topics: Analyzing Dispersion Effects Across Different Language Models

TL;DR

This paper investigates how writing style influences embedding dispersion in language models, revealing stylistic effects beyond topic modeling and enhancing understanding of model interpretability across languages.

Contribution

It introduces an analysis of stylistic effects on embedding dispersion in multiple language models, extending beyond traditional topic-based interpretations.

Findings

Style significantly affects embedding dispersion across models

Language models show different sensitivity to stylistic variations

Insights improve interpretability of language model representations

Abstract

This paper analyzes how writing style affects the dispersion of embedding vectors across multiple, state-of-the-art language models. While early transformer models primarily aligned with topic modeling, this study examines the role of writing style in shaping embedding spaces. Using a literary corpus that alternates between topics and styles, we compare the sensitivity of language models across French and English. By analyzing the particular impact of style on embedding dispersion, we aim to better understand how language models process stylistic information, contributing to their overall interpretability.