Large Deviations and Metastability Analysis for Heavy-Tailed Dynamical Systems

TL;DR

This paper develops a comprehensive large deviations framework for heavy-tailed stochastic systems, analyzing rare events, exit times, and paths, with applications to deep learning algorithms involving truncation mechanisms.

Contribution

It introduces novel large deviations principles for heavy-tailed systems with truncation, extending Freidlin-Wentzell theory to these complex stochastic dynamics.

Findings

Established sharp large deviations asymptotics for heavy-tailed processes

Characterized the distribution of exit times and locations in rare events

Unveiled phase transitions related to truncation thresholds

Abstract

This paper introduces novel frameworks for large deviations and metastability analysis in heavy-tailed stochastic dynamical systems. We develop and apply these frameworks within the context of stochastic difference equation $X^\eta_{j+1}(x) = X^\eta_{j}(x) + \eta a\big( X^\eta_{j}(x)\big) + \eta \sigma\big( X^\eta_{j}(x)\big)Z_{j+1}$ and its variation with truncated dynamics $X^{\eta|b}_{j+1}(x) = X^{\eta|b}_{j}( x) + \varphi_b\big(\eta a\big( X^{\eta|b}_{j}( x)\big) + \eta \sigma\big( X^{\eta|b}_{j}( x)\big) Z_{j+1}\big)$, where $\varphi_b(x) = (x/\|x\|)\max\{\|x\|, b\}$. The truncation operator $\varphi_b(\cdot)$ is often introduced as a modulation mechanism in heavy-tailed systems, such as stochastic gradient descent algorithms in deep learning. Thus, it is crucial to successfully analyze both $X^{\eta}_{j}(x)$ and $X^{\eta|b}_{j}(x)$. We establish locally uniform sample-path large deviations for both processes and translate these asymptotics into precise characterizations of the joint distributions of the first exit times and exit locations. Our large deviations asymptotics are sharp enough to rigorously characterize \emph{the catastrophe principle} by establishing the distributional limit of the sample paths conditional on the rare events of interest, thereby revealing the most likely paths through which rare events arise in heavy-tailed dynamical systems. Moreover the resulting limit theorem unveils a discrete hierarchy of phase transitions (i.e., exit times) as the truncation threshold $b$ varies. Together, these developments serve as a heavy-tailed counterpart of the classical Freidlin-Wentzell theory. We also present the corresponding results for continuous-time processes in the appendix.