Online Inference with Multi-modal Likelihood Functions

TL;DR

This paper introduces an online algorithm for parameter estimation in multi-modal likelihood functions, demonstrating convergence and robustness in linear regression, suitable for low to moderate dimensions.

Contribution

The paper presents a novel online inference algorithm that converges under multi-modal likelihoods and adapts to robust linear regression with outliers.

Findings

Convergence rate of the estimator is established under standard conditions.

The method is robust to outliers in linear regression models.

Applicable to low or moderate dimensional problems due to computational cost.

Abstract

Let $(Y_t)_{t\geq 1}$ be a sequence of i.i.d.\ observations and $\{f_\theta,\theta\in \mathbb{R}^d\}$ be a parametric model. We introduce a new online algorithm for computing a sequence $(\hat{\theta}_t)_{t\geq 1}$ which is shown to converge almost surely to $\text{argmax}_{\theta\in \mathbb{R}^d}\mathbb{E}[\log f_\theta(Y_1)]$ at rate $ \mathcal{O}(\log (t)^{(1+\varepsilon)/2}t^{-1/2})$, with $\varepsilon>0$ a user specified parameter. This convergence result is obtained under standard conditions on the statistical model and, most notably, we allow the mapping $\theta\mapsto \mathbb{E}[\log f_\theta(Y_1)]$ to be multi-modal. However, the computational cost to process each observation grows exponentially with the dimension of $\theta$, which makes the proposed approach applicable to low or moderate dimensional problems only. We also derive a version of the estimator $\hat{\theta}_t$ which is well suited to Student-t linear regression models. The corresponding estimator of the regression coefficients is robust to the presence of outliers, as shown by experiments on simulated and real data, and thus, as a by-product of this work, we obtain a new online and adaptive robust estimation method for linear regression models.