A Fast Incremental Gaussian Mixture Model

TL;DR

This paper introduces a faster, more scalable incremental Gaussian Mixture Model algorithm that reduces computational complexity from cubic to quadratic in data dimensions, enabling high-dimensional data processing.

Contribution

The authors derive formulas to work directly with precision matrices, significantly improving the scalability of incremental Gaussian mixture models for high-dimensional data.

Findings

Reduced complexity from O(NKD^3) to O(NKD^2)

Demonstrated effectiveness on high-dimensional classification datasets

Achieved faster incremental learning in high-dimensional spaces

Abstract

This work builds upon previous efforts in online incremental learning, namely the Incremental Gaussian Mixture Network (IGMN). The IGMN is capable of learning from data streams in a single-pass by improving its model after analyzing each data point and discarding it thereafter. Nevertheless, it suffers from the scalability point-of-view, due to its asymptotic time complexity of $\operatorname{O}\bigl(NKD^3\bigr)$ for $N$ data points, $K$ Gaussian components and $D$ dimensions, rendering it inadequate for high-dimensional data. In this paper, we manage to reduce this complexity to $\operatorname{O}\bigl(NKD^2\bigr)$ by deriving formulas for working directly with precision matrices instead of covariance matrices. The final result is a much faster and scalable algorithm which can be applied to high dimensional tasks. This is confirmed by applying the modified algorithm to high-dimensional classification datasets.