Convergent Block Coordinate Descent for Training Tikhonov Regularized Deep Neural Networks

TL;DR

This paper introduces a convergent block coordinate descent algorithm for training deep neural networks by transforming ReLU activations into a smooth multi-convex form, leading to better test errors on MNIST.

Contribution

It develops a novel smooth multi-convex formulation for DNN training and proves global convergence of the BCD algorithm with improved empirical performance.

Findings

BCD algorithm converges globally to a stationary point.

DNNs trained with BCD outperform SGD variants on MNIST.

The method ensures numerically stable convex subproblems.

Abstract

By lifting the ReLU function into a higher dimensional space, we develop a smooth multi-convex formulation for training feed-forward deep neural networks (DNNs). This allows us to develop a block coordinate descent (BCD) training algorithm consisting of a sequence of numerically well-behaved convex optimizations. Using ideas from proximal point methods in convex analysis, we prove that this BCD algorithm will converge globally to a stationary point with R-linear convergence rate of order one. In experiments with the MNIST database, DNNs trained with this BCD algorithm consistently yielded better test-set error rates than identical DNN architectures trained via all the stochastic gradient descent (SGD) variants in the Caffe toolbox.