Scalable and Communication-Efficient Varying Coefficient Mixed Effect Models: Methodology, Theory, and Applications

TL;DR

This paper introduces a scalable, communication-efficient framework for Varying Coefficient Mixed Models, enabling large-scale, distributed inference with theoretical guarantees and practical applications in human migration analysis.

Contribution

It develops a novel inference method for VCMMs that preserves likelihood information under communication constraints and supports scalable, stable estimation in large, distributed datasets.

Findings

The method achieves first-order efficiency with minimal communication.

Simulations confirm accuracy and scalability in large datasets.

Application to migration data reveals dynamic spatial patterns.

Abstract

Human migration exhibits complex spatiotemporal dependence driven by environmental and socioeconomic forces. Modeling such patterns at scale requires methods that accommodate many random effects while remaining feasible when raw data or large design matrices cannot be freely shared across distributed nodes. We develop a communication-efficient inference framework for Varying Coefficient Mixed Models (VCMMs) with flexible mean structures and large correlated random-effect components. Using a Bayesian hierarchical representation of penalized splines, we derive sufficient statistics that preserve each node's likelihood contribution and recover the estimator from the full data under unrestricted communication. Under communication constraints, these statistics support a one-step communication-efficient estimator with first-order efficiency. An SVD-enhanced implementation stabilizes large or ill-conditioned random-effect covariance operators. Theory establishes likelihood preservation, convergence, asymptotic efficiency, and finite-sample concentration. Simulations and U.S. migration-flow data demonstrate accuracy, scalability, and recovery of dynamic spatial patterns.