Deep Language Geometry: Constructing a Metric Space from LLM Weights

TL;DR

This paper presents a new method to construct a metric space of languages using internal weights of large language models, revealing linguistic relationships and evolution.

Contribution

It introduces a novel framework that derives language representations from LLM weights, capturing intrinsic linguistic features without manual feature engineering.

Findings

Aligns with known linguistic families

Reveals unexpected inter-language connections

Applicable across diverse datasets and multilingual LLMs

Abstract

We introduce a novel framework that utilizes the internal weight activations of modern Large Language Models (LLMs) to construct a metric space of languages. Unlike traditional approaches based on hand-crafted linguistic features, our method automatically derives high-dimensional vector representations by computing weight importance scores via an adapted pruning algorithm. Our approach captures intrinsic language characteristics that reflect linguistic phenomena. We validate our approach across diverse datasets and multilingual LLMs, covering 106 languages. The results align well with established linguistic families while also revealing unexpected inter-language connections that may indicate historical contact or language evolution. The source code, computed language latent vectors, and visualization tool are made publicly available at https://github.com/mshamrai/deep-language-geometry.