Improvements to Supervised EM Learning of Shared Kernel Models by Feature Space Partitioning

TL;DR

This paper enhances supervised EM learning for shared kernel models by rigorously deriving the algorithm and reducing computational complexity through feature space partitioning, enabling efficient training on higher-dimensional data.

Contribution

It provides a detailed EM derivation for the Gaussian shared kernel model and introduces feature space partitioning to improve efficiency and scalability.

Findings

Partitioned SKEM achieves lower complexity

Improved performance on MNIST dataset

Comparable or better accuracy than standard classifiers

Abstract

Expectation maximisation (EM) is usually thought of as an unsupervised learning method for estimating the parameters of a mixture distribution, however it can also be used for supervised learning when class labels are available. As such, EM has been applied to train neural nets including the probabilistic radial basis function (PRBF) network or shared kernel (SK) model. This paper addresses two major shortcomings of previous work in this area: the lack of rigour in the derivation of the EM training algorithm; and the computational complexity of the technique, which has limited it to low dimensional data sets. We first present a detailed derivation of EM for the Gaussian shared kernel model PRBF classifier, making use of data association theory to obtain the complete data likelihood, Baum's auxiliary function (the E-step) and its subsequent maximisation (M-step). To reduce complexity of the resulting SKEM algorithm, we partition the feature space into $R$ non-overlapping subsets of variables. The resulting product decomposition of the joint data likelihood, which is exact when the feature partitions are independent, allows the SKEM to be implemented in parallel and at $R^2$ times lower complexity. The operation of the partitioned SKEM algorithm is demonstrated on the MNIST data set and compared with its non-partitioned counterpart. It eventuates that improved performance at reduced complexity is achievable. Comparisons with standard classification algorithms are provided on a number of other benchmark data sets.