Training Deep Neural Networks via Optimization Over Graphs

TL;DR

This paper introduces a novel distributed optimization approach over graphs for training deep neural networks, utilizing ADMM to improve overfitting resistance compared to traditional methods like SGD and Adam.

Contribution

It reformulates neural network training as a graph-based optimization problem and applies ADMM for layerwise weight updates, offering an alternative to gradient-based methods.

Findings

ADMM-based training is less sensitive to overfitting than SGD and Adam.

The method effectively handles ReLU and DCutLU activations.

Empirical results demonstrate competitive training performance.

Abstract

In this work, we propose to train a deep neural network by distributed optimization over a graph. Two nonlinear functions are considered: the rectified linear unit (ReLU) and a linear unit with both lower and upper cutoffs (DCutLU). The problem reformulation over a graph is realized by explicitly representing ReLU or DCutLU using a set of slack variables. We then apply the alternating direction method of multipliers (ADMM) to update the weights of the network layerwise by solving subproblems of the reformulated problem. Empirical results suggest that the ADMM-based method is less sensitive to overfitting than the stochastic gradient descent (SGD) and Adam methods.