SGB: Stochastic Gradient Bound Method for Optimizing Partition Functions

TL;DR

This paper introduces a stochastic optimization method for partition functions, combining quadratic surrogates and second-order approximations, which outperforms standard SGD in experiments and has applications in deep learning.

Contribution

The paper proposes the Stochastic Partition Function Bound (SPFB), a novel stochastic optimization algorithm that uses quadratic surrogates and second-order information for efficient partition function optimization.

Findings

SPFB significantly outperforms SGD in logistic regression experiments.

The method has a proven sub-linear convergence rate.

A low-rank variant, LSPFB, is also constructed for efficiency.

Abstract

This paper addresses the problem of optimizing partition functions in a stochastic learning setting. We propose a stochastic variant of the bound majorization algorithm that relies on upper-bounding the partition function with a quadratic surrogate. The update of the proposed method, that we refer to as Stochastic Partition Function Bound (SPFB), resembles scaled stochastic gradient descent where the scaling factor relies on a second order term that is however different from the Hessian. Similarly to quasi-Newton schemes, this term is constructed using the stochastic approximation of the value of the function and its gradient. We prove sub-linear convergence rate of the proposed method and show the construction of its low-rank variant (LSPFB). Experiments on logistic regression demonstrate that the proposed schemes significantly outperform SGD. We also discuss how to use quadratic partition function bound for efficient training of deep learning models and in non-convex optimization.