Feed-Forward Neural Networks as a Mixed-Integer Program

TL;DR

This paper explores modeling ReLU neural networks as mixed-integer programs and investigates their use in training and evaluating neural network architectures, including binary and binarized models, through experiments on digit classification.

Contribution

It introduces a novel MIP formulation for trained ReLU neurons and applies it to train and analyze various neural network architectures, including binary and binarized DNNs.

Findings

MIP formulations effectively model ReLU neurons in neural networks.

The approach improves training processes for certain neural network architectures.

Experimental results on digit classification demonstrate the method's potential.

Abstract

Deep neural networks (DNNs) are widely studied in various applications. A DNN consists of layers of neurons that compute affine combinations, apply nonlinear operations, and produce corresponding activations. The rectified linear unit (ReLU) is a typical nonlinear operator, outputting the max of its input and zero. In scenarios like max pooling, where multiple input values are involved, a fixed-parameter DNN can be modeled as a mixed-integer program (MIP). This formulation, with continuous variables representing unit outputs and binary variables for ReLU activation, finds applications across diverse domains. This study explores the formulation of trained ReLU neurons as MIP and applies MIP models for training neural networks (NNs). Specifically, it investigates interactions between MIP techniques and various NN architectures, including binary DNNs (employing step activation functions) and binarized DNNs (with weights and activations limited to $-1,0,+1$). The research focuses on training and evaluating proposed approaches through experiments on handwritten digit classification models. The comparative study assesses the performance of trained ReLU NNs, shedding light on the effectiveness of MIP formulations in enhancing training processes for NNs.