Training Neural Networks with an algorithm for piecewise linear functions

TL;DR

This paper explores training neural networks using the Volume Algorithm, comparing its performance with Momentum, Adam, and COCOB algorithms to evaluate effectiveness.

Contribution

It introduces the application of the Volume Algorithm for neural network training and provides comparative analysis with popular optimization algorithms.

Findings

Volume Algorithm performs competitively with Momentum and Adam.

COCOB shows promising results in certain scenarios.

The study offers insights into alternative optimization methods for neural networks.

Abstract

We present experiments on training neural networks with an algorithm that was originally designed as a subgradient method, namely the Volume Algorithm. We compare with two of the most frequently used algorithms like Momentum and Adam. We also compare with the newly proposed algorithm COCOB.