A Latent Variable Approach to Learning High-dimensional Multivariate longitudinal Data

TL;DR

This paper introduces a latent variable model for high-dimensional multivariate longitudinal data, enabling better inference and prediction of outcomes over time, especially with mixed data types and missing observations.

Contribution

It develops a novel latent variable framework that accounts for complex dependencies and handles mixed and incomplete data, with theoretical inference tools and an application to shopping behavior.

Findings

Established a central limit theorem for regression coefficients.

Introduced an information criterion for selecting the number of factors.

Applied the model successfully to real-world grocery shopping data.

Abstract

High-dimensional multivariate longitudinal data, which arise when many outcome variables are measured repeatedly over time, are becoming increasingly common in social, behavioral and health sciences. We propose a latent variable model for drawing statistical inferences on covariate effects and predicting future outcomes based on high-dimensional multivariate longitudinal data. This model introduces unobserved factors to account for the between-variable and across-time dependence and assist the prediction. Statistical inference and prediction tools are developed under a general setting that allows outcome variables to be of mixed types and possibly unobserved for certain time points, for example, due to right censoring. A central limit theorem is established for drawing statistical inferences on regression coefficients. Additionally, an information criterion is introduced to choose the number of factors. The proposed model is applied to customer grocery shopping records to predict and understand shopping behavior.