From the New World of Word Embeddings: A Comparative Study of Small-World Lexico-Semantic Networks in LLMs

TL;DR

This study constructs and analyzes lexico-semantic networks from input embeddings of various large language models, revealing small-world properties and how network complexity scales with model size, offering insights into their semantic structures.

Contribution

It introduces a novel method to build and compare lexico-semantic networks from decoder-only LLMs, highlighting their small-world characteristics and semantic richness across different scales.

Findings

Networks exhibit small-world properties with high clustering and short paths.

Larger LLMs produce more complex networks with less pronounced small-world effects.

Semantic relations from WordNet and cross-lingual analyses validate the networks' meaningful structure.

Abstract

Lexico-semantic networks represent words as nodes and their semantic relatedness as edges. While such networks are traditionally constructed using embeddings from encoder-based models or static vectors, embeddings from decoder-only large language models (LLMs) remain underexplored. Unlike encoder models, LLMs are trained with a next-token prediction objective, which does not directly encode the meaning of the current token. In this paper, we construct lexico-semantic networks from the input embeddings of LLMs with varying parameter scales and conduct a comparative analysis of their global and local structures. Our results show that these networks exhibit small-world properties, characterized by high clustering and short path lengths. Moreover, larger LLMs yield more intricate networks with less small-world effects and longer paths, reflecting richer semantic structures and relations. We further validate our approach through analyses of common conceptual pairs, structured lexical relations derived from WordNet, and a cross-lingual semantic network for qualitative words.