# Quantifying Long Range Dependence in Language and User Behavior to improve RNNs

## TL;DR

This paper introduces a scalable method to quantify Long Range Dependence in sequential data, revealing its presence in language and user behavior, and leveraging this insight to improve RNN architectures for language and recommendation tasks.

## Contribution

It proposes a novel, model-independent estimation procedure for LRD in large-scale sequential datasets and demonstrates its utility in designing more effective RNN models.

## Key findings

- LRD is reliably measurable in large-scale language and user behavior data.
- Measuring LRD improves RNN design, leading to state-of-the-art results.
- Evolutive RNNs outperform traditional models in language understanding and recommendation tasks.

## Abstract

Characterizing temporal dependence patterns is a critical step in understanding the statistical properties of sequential data. Long Range Dependence (LRD) --- referring to long-range correlations decaying as a power law rather than exponentially w.r.t. distance --- demands a different set of tools for modeling the underlying dynamics of the sequential data. While it has been widely conjectured that LRD is present in language modeling and sequential recommendation, the amount of LRD in the corresponding sequential datasets has not yet been quantified in a scalable and model-independent manner. We propose a principled estimation procedure of LRD in sequential datasets based on established LRD theory for real-valued time series and apply it to sequences of symbols with million-item-scale dictionaries. In our measurements, the procedure estimates reliably the LRD in the behavior of users as they write Wikipedia articles and as they interact with YouTube. We further show that measuring LRD better informs modeling decisions in particular for RNNs whose ability to capture LRD is still an active area of research. The quantitative measure informs new Evolutive Recurrent Neural Networks (EvolutiveRNNs) designs, leading to state-of-the-art results on language understanding and sequential recommendation tasks at a fraction of the computational cost.