Empirical study of PROXTONE and PROXTONE$^+$ for Fast Learning of Large Scale Sparse Models

TL;DR

This paper introduces PROXTONE$^+$, a hybrid optimization method combining PROXTONE and first-order techniques to efficiently train large-scale sparse neural networks, achieving faster convergence and significant model size reduction.

Contribution

It proposes a novel hybrid training approach, PROXTONE$^+$, that accelerates convergence and enhances sparsity in large-scale neural network training.

Findings

PROXTONE and PROXTONE$^+$ double convergence speed.

PROXTONE$^+$ reduces model size to 0.5%.

Both methods outperform traditional first-order methods.

Abstract

PROXTONE is a novel and fast method for optimization of large scale non-smooth convex problem \cite{shi2015large}. In this work, we try to use PROXTONE method in solving large scale \emph{non-smooth non-convex} problems, for example training of sparse deep neural network (sparse DNN) or sparse convolutional neural network (sparse CNN) for embedded or mobile device. PROXTONE converges much faster than first order methods, while first order method is easy in deriving and controlling the sparseness of the solutions. Thus in some applications, in order to train sparse models fast, we propose to combine the merits of both methods, that is we use PROXTONE in the first several epochs to reach the neighborhood of an optimal solution, and then use the first order method to explore the possibility of sparsity in the following training. We call such method PROXTONE plus (PROXTONE$^+$). Both PROXTONE and PROXTONE$^+$ are tested in our experiments, and which demonstrate both methods improved convergence speed twice as fast at least on diverse sparse model learning problems, and at the same time reduce the size to 0.5\% for DNN models. The source of all the algorithms is available upon request.