Scalable solution to crossed random effects model with random slopes

TL;DR

This paper introduces scalable variational EM algorithms for crossed random effects models with random slopes, enabling efficient analysis of large datasets in fields like psychology and e-commerce.

Contribution

It extends existing scalable methods to include random slopes and covariance estimation, addressing a key computational challenge.

Findings

Proposed algorithms are significantly faster than standard methods on large datasets.

The method outperforms ordinary least squares in estimating sampling uncertainty.

Demonstrated effectiveness on datasets with millions of observations.

Abstract

The crossed random effects model is widely used, finding applications in various fields such as longitudinal studies, e-commerce, and recommender systems, among others. However, these models encounter scalability challenges, as the computational time for standard algorithms grows superlinearly with the number N of observations in the data set, commonly $\Omega(N^{3/2})$ or worse. Recent published works present scalable methods for crossed random effects in linear models and some generalized linear models, but those methods only allow for random intercepts. In this paper, we devise scalable algorithms for models that include random slopes. This addition brings substantial difficulty in estimating the random-effect covariance matrices in a scalable way. We address this issue by using a variational EM algorithm. Our proposed approach accommodates both diagonal covariance matrices and cases where no structure is assumed-a scenario common in fields such as psychology and neuroscience. In simulations, the proposed method is substantially faster than standard methods for large $N$. It is also more efficient than ordinary least squares which has a problem of greatly underestimating the sampling uncertainty in parameter estimates. We illustrate the new method on a MovieLens dataset, as well as a large data set (five million observations) from the online retailer Stitch Fix.