Backfitting for large scale crossed random effects regressions

TL;DR

This paper introduces a backfitting algorithm for large-scale crossed random effects regressions that achieves linear computational cost, relaxing previous strict conditions and demonstrating practical efficiency on real data.

Contribution

It proposes a novel backfitting method for generalized least squares in crossed random effects models with proven linear complexity and relaxed assumptions.

Findings

Algorithm costs O(N) in practice.

Conditions for convergence are less strict than previous methods.

Effective on real-world ratings data.

Abstract

Regression models with crossed random effect errors can be very expensive to compute. The cost of both generalized least squares and Gibbs sampling can easily grow as $N^{3/2}$ (or worse) for $N$ observations. Papaspiliopoulos et al. (2020) present a collapsed Gibbs sampler that costs $O(N)$, but under an extremely stringent sampling model. We propose a backfitting algorithm to compute a generalized least squares estimate and prove that it costs $O(N)$. A critical part of the proof is in ensuring that the number of iterations required is $O(1)$ which follows from keeping a certain matrix norm below $1-\delta$ for some $\delta>0$. Our conditions are greatly relaxed compared to those for the collapsed Gibbs sampler, though still strict. Empirically, the backfitting algorithm has a norm below $1-\delta$ under conditions that are less strict than those in our assumptions. We illustrate the new algorithm on a ratings data set from Stitch Fix.