# Long-Range Dependence in Word Time Series: The Cosine Correlation of Embeddings

## TL;DR

This paper explores how word usage in texts shows long-term memory patterns, comparing human writing to AI-generated text.

## Contribution

The study introduces a novel method using cosine correlation of word embeddings to detect long-range dependence in texts.

## Key findings

- Cosine correlation of word2vec embeddings shows stretched exponential decay in human texts up to 1000-word lags.
- Large language models do not exhibit systematic long-range dependence in their generated texts.
- The findings suggest a need for memory-rich architectures beyond Transformers and hidden Markov models.

## Abstract

We analyze long-range dependence (LRD) for word time series, understood as a slower than exponential decay of the two-point Shannon mutual information. We achieve this by examining the decay of the cosine correlation, a proxy object defined in terms of the cosine similarity between word2vec embeddings of two words, computed by an analogy to the Pearson correlation. By the Pinsker inequality, the squared cosine correlation between two random vectors lower bounds the mutual information between them. Using the Standardized Project Gutenberg Corpus, we find that the cosine correlation between word2vec embeddings exhibits a readily visible stretched exponential decay for lags roughly up to 1000 words, thus corroborating the presence of LRD. By contrast, for the Human vs. LLM Text Corpus entailing texts generated by large language models, there is no systematic signal of LRD. Our findings may support the need for novel memory-rich architectures in large language models that exceed not only hidden Markov models but also Transformers.