Non-conjugate variational Bayes for pseudo-likelihood mixed effect models

TL;DR

This paper introduces a simple, unified non-conjugate variational Bayes algorithm for a broad class of Bayesian mixed effect models, effectively handling non-conjugate priors and non-smooth loss functions.

Contribution

It presents a novel message passing optimization strategy with closed-form updates for popular models, improving computational efficiency and approximation accuracy.

Findings

Demonstrates effectiveness through theoretical analysis.

Shows superior computational efficiency in experiments.

Achieves accurate posterior approximations for complex models.

Abstract

We propose a unified, yet simple to code, non-conjugate variational Bayes algorithm for posterior approximation of generic Bayesian generalized mixed effect models. Specifically, we consider regression models identified by a linear predictor, eventually transformed using a bijective link, where the prediction misfit is measured using, possibly non-differentiable, loss functions. Examples include generalized linear models, quasi-likelihood models, and robust regression. To address the limitations of non-conjugate settings, we employ an efficient message passing optimization strategy under a Gaussian variational approximation of the posterior. The resulting algorithms automatically account for non-conjugate priors and non-smooth losses, without requiring model-specific data-augmented representations. Besides the general formulation, we provide closed-form updates for popular model specifications, including quantile regression and support vector machines. Overall, theoretical and empirical results highlight the effectiveness of the proposed method, demonstrating its computational efficiency and approximation accuracy as an alternative to existing Bayesian techniques.