High-Dimensional Statistics: Reflections on Progress and Open Problems

TL;DR

This paper reviews two decades of progress in high-dimensional statistics, emphasizing recent advances, challenges, and open problems across various scientific domains and theoretical connections.

Contribution

It synthesizes key developments, highlights common themes, and identifies open problems in high-dimensional statistics over the past twenty years.

Findings

Significant progress driven by technological advances in data collection.

Complex modern datasets challenge traditional statistical methods.

Deep connections established with optimization, random matrix theory, and computer science.

Abstract

Over the past two decades, the field of high-dimensional statistics has experienced substantial progress, driven largely by technological advances that have dramatically reduced the cost and effort for data collection and storage across a broad range of domains, including biology, medicine, astronomy, and the social and environmental sciences. Modern datasets are increasingly complex, often exhibiting rich dependency, heterogeneity, and other features that challenge traditional statistical methods. In response, high-dimensional statistics has evolved to address more sophisticated estimation and inference problems. This evolution has, in turn, fostered deep connections with and contributions to a wide range of research areas, including optimization, concentration of measure, random matrix theory, information theory, and theoretical computer science. Given the rapid pace of recent developments in high-dimensional statistics, our goal is to synthesize representative advances, highlight common themes and open problems, and point to important works that offer entry points into the field.