Elastic regularization in restricted Boltzmann machines: Dealing with $p\gg N$

TL;DR

This paper introduces the elastic restricted Boltzmann machine (eRBM), a new model designed to effectively handle the high-dimensional $p extgreater N$ data scenarios common in computational biology, especially cancer data analysis.

Contribution

The paper proposes the eRBM model with elastic regularization, providing theoretical analysis and demonstrating efficient training via contrastive divergence, addressing the $p extgreater N$ challenge.

Findings

eRBM outperforms traditional RBMs in high-dimensional settings.

The elastic regularization improves model stability and accuracy.

Efficient training is achieved through classic contrastive divergence.

Abstract

Restricted Boltzmann machines (RBMs) are endowed with the universal power of modeling (binary) joint distributions. Meanwhile, as a result of their confining network structure, training RBMs confronts less difficulties (compared with more complicated models, e.g., Boltzmann machines) when dealing with approximation and inference issues. However, in certain computational biology scenarios, such as the cancer data analysis, employing RBMs to model data features may lose its efficacy due to the "$p\gg N$" problem, in which the number of features/predictors is much larger than the sample size. The "$p\gg N$" problem puts the bias-variance trade-off in a more crucial place when designing statistical learning methods. In this manuscript, we try to address this problem by proposing a novel RBM model, called elastic restricted Boltzmann machine (eRBM), which incorporates the elastic regularization term into the likelihood/cost function. We provide several theoretical analysis on the superiority of our model. Furthermore, attributed to the classic contrastive divergence (CD) algorithm, eRBMs can be trained efficiently. Our novel model is a promising method for future cancer data analysis.