Online Heterogeneous Mixture Learning for Big Data

TL;DR

This paper introduces an online learning approach for big data analysis that handles heterogeneity, demonstrating rapid convergence to batch-level accuracy through experiments.

Contribution

It presents a novel online heterogeneous mixture learning method that achieves comparable accuracy to batch methods with faster convergence.

Findings

Online method converges quickly to batch accuracy.

Achieves comparable accuracy to traditional batch learning.

Effective for big data heterogeneity.

Abstract

We propose the online machine learning for big data analysis with heterogeneity. We performed an experiment to compare the accuracy of each iteration between batch one and online one. It is possible to converge quickly with the same accuracy as the batch one.

Tables

Table 1. TABLE I: data

the number of data	10,000
the number of component mixture	4
mixing coefficient	0.1,0.2,0.3,0.4
means, convariances	random
the number of dimensions	10

Equations

$$\gamma {\left(z_{nk}\right)}^{(t+1)}=\frac{{\pi }_k^{(t)}N\left({\mathbf{x}}_n|{\mathbf{\mu }}_k^{(t)},{\mathbf{\Sigma }}_k^{(t)}\right)}{{\sum }_{j=1}^K{\pi }_j^{(t)}N\left({\mathbf{x}}_n|{\mathbf{\mu }}_j^{(t)},{\mathbf{\Sigma }}_j^{(t)}\right)},(k=1,\dots ,K)$$

$$s_{nk}^{(t+1)}=\gamma {\left(z_{nk}\right)}^{(t+1)}-\gamma {\left(z_{nk}\right)}^{(t)}$$

$$N_k^{(t+1)}=\sum _{n=1}^N\gamma {\left(z_{nk}\right)}^{(t)}+s_{nk}^{(t+1)}=N_k^{(t)}+s_{nk}^{(t)}$$

$${\pi }_k^{(t+1)}={\pi }_k^{(t)}+\frac{s_{nk}^{(t+1)}}{N}$$

$${\mu }_k^{(t+1)}={\mu }_k^{(t)}+\frac{s_{nk}^{(t+1)}}{N_k^{(t+1)}}\left({\mathbf{x}}_n-{\mathbf{\mu }}_k^{(t)}\right)$$

$${\mathbf{\Sigma }}_k^{(t+1)}$$

$$\left\{{\mathbf{\Sigma }}_k^{(t)}+\frac{s_{nk}^{(t+1)}}{N_k^{(t+1)}}\left({\mathbf{x}}_n-{\mathbf{\mu }}_k^{(t)}\right){\left({\mathbf{x}}_n-{\mathbf{\mu }}_k^{(t)}\right)}^T\right\}$$

$$FI{C}_{\text{online}}\left(x_n,M\right)$$

$$=\underset{q}{\max }\left\{J_{\text{online}}\left(q,\overline{\theta },x_n\right)\right\}$$

$$J_{\text{ online }}\left(q,\overline{\theta },x_n\right)$$

$$+q\left(z_{nc}\right)[\log p\left(x_n,z_{nc}|\overline{\theta }\right)$$

$$-\frac{1}{2}\log N-\sum _{c=1}^C\frac{D_c}{2}\left\{\log z_{nc}-\log q\left(z_{nc}\right)\right\}]$$

Full text

I Introduction

There is a kind of heterogeneous mixture learning for big data analysis with heterogeneity. This is batch learning using a batch EM algorithm for model generation[1]. Therefore, we use the incremental EM algorithm [2,3,4] which is an online EM algorithm to propose online heterogeneous mixture learning. Online heterogeneous mixture learning is possible to converge faster than the batch type with the same accuracy.

II online heterogeneous mixed learning

We propose online learning of heterogeneous mixed learning using the online method of EM algorithm for mixture of Gaussian. First of all, we introduce the incremental EM algorithm[2,3].

$E_{incremental}Step:$ We fix the parameters, and calculate the responsibility $\gamma$ and the amount of change in the responsibility $s_{k}$. we update the responsibility for one data $x_{n}$ with observation data $x_{N}$.

[TABLE]

We calculate the amount of change in the responsibility $s_{nk}$.

[TABLE]

[TABLE]

$M_{incremental}Step:$ We fix the esponsibility $\gamma(z_{nk})$ and the amount of change in the responsibility $s_{nk}$, and update each parameter.

[TABLE]

[TABLE]

[TABLE]

The crucial points of heterogeneous mixed learning are a factorized information criterion (FIC) and factorized asymptotic Bayesian inference (FAB)[1]. We have to make these available online. First, we improve FIC, which is metric of the model. Second, we improve FAB in response to change of FIC.

The $FIC_{online}$ which supports online learning is shown below.

[TABLE]

[TABLE]

It is not possible to evaluate $FIC_{online}$ directly because the parameters can not be determined analytically. In order to evaluate FIC, FAB maximizes an asymptotically-consistent lower bound of FIC. For updates incrementally, we improve FAB using the variation of the variational probability of the latent variable.

We calculate sequentially by repeating the following two steps $t$ times.

$V_{online}Step:$ We optimize the distribution $q(z_{nc})$ of latent variables ${z}^{N}$, and calculate the distribution of latent variables and their variation $s_{nc}$ for additional data $x_{n}$.

$M_{online}Step:$ We optimize components of mixture of Gaussian and parameters $\theta$.

III results of experiment

We compare the results of conventional batch heterogeneous mixture learning [1] and online heterogeneous mixture learning which is proposed in this paper in the same environment and conditions.

In this experiment, the data used for learning is normal random number generated from the mixture of Gaussian. The mixture of Gaussian needs three parameters which are means, convariances and mixing coefficient. We specified these three parameters and the number of dimensions, and we made the dataset for this experiment. TABLE 1 show details.

We measured how the FIC changed with each iteration to compare the convergence speed of online learning with it of batch learning. The number of iterations until convergence was also included in the evaluation. The experiment is performed 10 times, and the average is taken as the experimental result. We experimented by changing the number of data [500, 10000] (Fig. 1) and changing the number of dimensions [2, 4, 20] (Fig. 2). We experimented with the other parameters fixed.

In terms of the small number of iterations, online learning is better than batch learning for all diagrams. When comparing online and batch algorithms, the batch algorithm usually converges faster. However, in the EM algorithm, the on-line algorithm converges faster than the batch algorithm.

IV CONCLUSION

We proposed the online heterogeneous mixture learning for the purpose of speeding up the convergence of machine learning for heterogeneous data. It can be learned to the same accuracy of the batch one with fewer iterations than the batch one. It is also necessary to consider using the Stepwise EM algorithm [4] in which the work area is scalable.