Stochastic Descent Analysis of Representation Learning Algorithms

TL;DR

This paper introduces a new stochastic approximation theorem with verifiable assumptions, enabling rigorous analysis and design of various deep learning algorithms such as adaptive learning and contrastive divergence.

Contribution

It provides a novel stochastic approximation theorem tailored for state-dependent noise, facilitating theoretical understanding of key deep learning algorithms.

Findings

The theorem applies to adaptive learning algorithms.

It enables analysis of contrastive divergence learning.

Supports design of more reliable deep learning methods.

Abstract

Although stochastic approximation learning methods have been widely used in the machine learning literature for over 50 years, formal theoretical analyses of specific machine learning algorithms are less common because stochastic approximation theorems typically possess assumptions which are difficult to communicate and verify. This paper presents a new stochastic approximation theorem for state-dependent noise with easily verifiable assumptions applicable to the analysis and design of important deep learning algorithms including: adaptive learning, contrastive divergence learning, stochastic descent expectation maximization, and active learning.