Information-theoretic limits and approximate message-passing for high-dimensional time series

TL;DR

This paper explores the fundamental information-theoretic limits and evaluates the performance of the VAMP algorithm in high-dimensional, non-sparse time series models, providing new theoretical insights and empirical evidence of near-optimal inference.

Contribution

It derives a single-letter formula for mutual information and MMSE in high-dimensional time series without sparsity assumptions, and empirically demonstrates VAMP's robustness and near-optimality.

Findings

Derived a formula for mutual information and MMSE in the model.

VAMP performs robustly and often near-optimally empirically.

Extended analysis beyond sparse regimes to general high-dimensional time series.

Abstract

High-dimensional time series appear in many scientific setups, demanding a nuanced approach to model and analyze the underlying dependence structure. Theoretical advancements so far often rely on stringent assumptions regarding the sparsity of the underlying signal. In non-sparse regimes, analyses have primarily focused on linear regression models with the design matrix having independent rows. In this paper, we expand the scope by investigating a high-dimensional time series model wherein the number of features grows proportionally to the number of sampling points, without assuming sparsity in the signal. Specifically, we consider the stochastic regression model and derive a single-letter formula for the normalized mutual information between observations and the signal, as well as for minimum mean-square errors. We also empirically study the vector approximate message passing VAMP algorithm and show that, despite the lack of theoretical guarantees, its performance for inference in our time series model is robust and often statistically optimal.