The flow of ideas in word embeddings

TL;DR

This paper explores the flow of ideas in word embeddings using microrheology-inspired random walks, revealing anomalous diffusion patterns that relate to creativity and idea integration in language models.

Contribution

It introduces a novel approach combining microrheology and machine learning to analyze idea flow in word embeddings through similarity-based random walks.

Findings

Random walks in embeddings show signatures of anomalous diffusion.

The method quantifies how diverse ideas are incorporated in documents.

Potential links between diffusion patterns and creativity are suggested.

Abstract

The flow of ideas has been extensively studied by physicists, psychologists, and machine learning engineers. This paper adopts specific tools from microrheology to investigate the similarity-based flow of ideas. We introduce a random walker in word embeddings and study its behavior. Such similarity-mediated random walks through the embedding space show signatures of anomalous diffusion commonly observed in complex structured systems such as biological cells and complex fluids. The paper concludes by proposing the application of popular tools employed in the study of random walks and diffusion of particles under Brownian motion to assess quantitatively the incorporation of diverse ideas in a document. Overall, this paper presents a self-referenced method combining microrheology and machine learning concepts to explore the meandering tendencies of language models and their potential association with creativity.