Inferring serial correlation with dynamic backgrounds

TL;DR

This paper introduces a novel method for inferring serial correlation in sequential data with dynamic backgrounds, using a total variation constrained estimator and hypothesis testing, applicable in neuroscience, psychology, and econometrics.

Contribution

It proposes a new total variation constrained least squares estimator with hypothesis testing for serial correlation inference amidst unknown dynamic backgrounds, demonstrating near-optimal performance.

Findings

The method accurately infers serial correlation in complex dynamic backgrounds.

The estimator achieves near-optimal error bounds as shown by theoretical analysis.

Numerical and real data applications confirm superior performance over existing methods.

Abstract

Sequential data with serial correlation and an unknown, unstructured, and dynamic background is ubiquitous in neuroscience, psychology, and econometrics. Inferring serial correlation for such data is a fundamental challenge in statistics. We propose a total variation constrained least square estimator coupled with hypothesis tests to infer the serial correlation in the presence of unknown and unstructured dynamic background. The total variation constraint on the dynamic background encourages a piece-wise constant structure, which can approximate a wide range of dynamic backgrounds. The tuning parameter is selected via the Ljung-Box test to control the bias-variance trade-off. We establish a non-asymptotic upper bound for the estimation error through variational inequalities. We also derive a lower error bound via Fano's method and show the proposed method is near-optimal. Numerical simulation and a real study in psychology demonstrate the excellent performance of our proposed method compared with the state-of-the-art.