Efficient and Robust Mixed-Integer Optimization Methods for Training Binarized Deep Neural Networks

TL;DR

This paper introduces efficient mixed-integer optimization techniques for training binarized deep neural networks, including a global optimality approach, heuristics for efficiency, and a robustness model, demonstrating competitive performance on resource-limited devices.

Contribution

It presents a novel mixed-integer programming formulation for BDNNs, along with heuristics and a robustness model, advancing training methods for resource-efficient neural networks.

Findings

BDNNs can be globally optimized using mixed-integer linear programming.

Heuristics significantly reduce computational complexity.

BDNNs often outperform classical DNNs on small architectures.

Abstract

Compared to classical deep neural networks its binarized versions can be useful for applications on resource-limited devices due to their reduction in memory consumption and computational demands. In this work we study deep neural networks with binary activation functions and continuous or integer weights (BDNN). We show that the BDNN can be reformulated as a mixed-integer linear program with bounded weight space which can be solved to global optimality by classical mixed-integer programming solvers. Additionally, a local search heuristic is presented to calculate locally optimal networks. Furthermore to improve efficiency we present an iterative data-splitting heuristic which iteratively splits the training set into smaller subsets by using the k-mean method. Afterwards all data points in a given subset are forced to follow the same activation pattern, which leads to a much smaller number of integer variables in the mixed-integer programming formulation and therefore to computational improvements. Finally for the first time a robust model is presented which enforces robustness of the BDNN during training. All methods are tested on random and real datasets and our results indicate that all models can often compete with or even outperform classical DNNs on small network architectures confirming the viability for applications having restricted memory or computing power.