# Unifying the Stochastic Spectral Descent for Restricted Boltzmann Machines with Bernoulli or Gaussian Inputs

## TL;DR

This paper extends stochastic spectral descent (SSD) to Gaussian RBMs, enabling efficient training on continuous data by deriving new bounds and demonstrating improved convergence over traditional stochastic gradient descent.

## Contribution

The paper generalizes SSD to Gaussian RBMs, providing theoretical bounds and empirical evidence of faster convergence for continuous data modeling.

## Key findings

- SSD outperforms SGD in convergence speed
- Derived upper bounds for the partition function using Schatten-infinity norm
- Extended SSD applicability from binary to Gaussian RBMs

## Abstract

Stochastic gradient descent based algorithms are typically used as the general optimization tools for most deep learning models. A Restricted Boltzmann Machine (RBM) is a probabilistic generative model that can be stacked to construct deep architectures. For RBM with Bernoulli inputs, non-Euclidean algorithm such as stochastic spectral descent (SSD) has been specifically designed to speed up the convergence with improved use of the gradient estimation by sampling methods. However, the existing algorithm and corresponding theoretical justification depend on the assumption that the possible configurations of inputs are finite, like binary variables. The purpose of this paper is to generalize SSD for Gaussian RBM being capable of mod- eling continuous data, regardless of the previous assumption. We propose the gradient descent methods in non-Euclidean space of parameters, via de- riving the upper bounds of logarithmic partition function for RBMs based on Schatten-infinity norm. We empirically show that the advantage and improvement of SSD over stochastic gradient descent (SGD).