Affine-invariant ensemble transform methods for logistic regression

TL;DR

This paper introduces affine-invariant ensemble transform methods for Bayesian logistic regression, extending ensemble Kalman and feedback particle filters to handle cross entropy loss, with gradient-free implementation and numerical validation.

Contribution

It develops affine-invariant ensemble transform algorithms for Bayesian logistic regression, incorporating gradient-free and subsampling techniques, and introduces an SDE-based sampling method.

Findings

Methods are affine-invariant and gradient-free for nonlinear cases

Algorithms perform well in numerical experiments

Approach extends ensemble Kalman filter to cross entropy loss

Abstract

We investigate the application of ensemble transform approaches to Bayesian inference of logistic regression problems. Our approach relies on appropriate extensions of the popular ensemble Kalman filter and the feedback particle filter to the cross entropy loss function and is based on a well-established homotopy approach to Bayesian inference. The arising finite particle evolution equations as well as their mean-field limits are affine-invariant. Furthermore, the proposed methods can be implemented in a gradient-free manner in case of nonlinear logistic regression and the data can be randomly subsampled similar to mini-batching of stochastic gradient descent. We also propose a closely related SDE-based sampling method which again is affine-invariant and can easily be made gradient-free. Numerical examples demonstrate the appropriateness of the proposed methodologies.