ReLU activated Multi-Layer Neural Networks trained with Mixed Integer Linear Programs

TL;DR

This paper demonstrates that ReLU-activated multi-layer neural networks can be trained using Mixed Integer Linear Programs, achieving comparable accuracy to traditional methods on MNIST.

Contribution

It introduces a novel MILP-based training method for neural networks, providing an alternative to gradient-based optimization.

Findings

Achieved MNIST accuracy comparable to TensorFlow/Keras.

Validated MILP-based training on simple neural networks.

Showed iterative layer-wise weight adjustment using MILPs.

Abstract

In this paper, it is demonstrated through a case study that multilayer feedforward neural networks activated by ReLU functions can in principle be trained iteratively with Mixed Integer Linear Programs (MILPs) as follows. Weights are determined with batch learning. Multiple iterations are used per batch of training data. In each iteration, the algorithm starts at the output layer and propagates information back to the first hidden layer to adjust the weights using MILPs or Linear Programs. For each layer, the goal is to minimize the difference between its output and the corresponding target output. The target output of the last (output) layer is equal to the ground truth. The target output of a previous layer is defined as the adjusted input of the following layer. For a given layer, weights are computed by solving a MILP. Then, except for the first hidden layer, the input values are also modified with a MILP to better match the layer outputs to their corresponding target outputs. The method was tested and compared with Tensorflow/Keras (Adam optimizer) using two simple networks on the MNIST dataset containing handwritten digits. Accuracies of the same magnitude as with Tensorflow/Keras were achieved.