Dynamic Mixture of Experts Models for Online Prediction

TL;DR

This paper introduces a dynamic mixture of experts model with time-evolving parameters, employing a sequential Monte Carlo algorithm for online inference, applicable to real-time prediction tasks such as software fault detection.

Contribution

It extends mixture of experts models by allowing parameters to evolve over time and develops a novel online inference algorithm for such models.

Findings

Effective in simulated data scenarios

Successfully applied to industrial software fault prediction

Handles both static and dynamic parameters

Abstract

A mixture of experts models the conditional density of a response variable using a mixture of regression models with covariate-dependent mixture weights. We extend the finite mixture of experts model by allowing the parameters in both the mixture components and the weights to evolve in time by following random walk processes. Inference for time-varying parameters in richly parameterized mixture of experts models is challenging. We propose a sequential Monte Carlo algorithm for online inference and based on a tailored proposal distribution built on ideas from linear Bayes methods and the EM algorithm. The method gives a unified treatment for mixtures with time-varying parameters, including the special case of static parameters. We assess the properties of the method on simulated data and on industrial data where the aim is to predict software faults in a continuously upgraded large-scale software project.