Pruning for efficient deterministic global optimization over trained ReLU neural networks

TL;DR

This paper demonstrates that pruning neural networks significantly accelerates embedded optimization problems by reducing problem size and tightening bounds, enabling faster solutions in complex engineering applications.

Contribution

It provides a theoretical and empirical analysis of how weight and node pruning improve the efficiency of ReLU neural network-based optimization models.

Findings

Pruning achieves speedups of up to 10^3-10^4 times.

Pruning reduces problem size and number of integer variables.

Pruning tightens big-M bounds, improving solver performance.

Abstract

Neural networks are increasingly used as surrogates in optimization problems to replace computationally expensive models. However, embedding ReLU neural networks in mathematical programs introduces significant computational challenges, particularly for deep and wide networks, due to both the formulation of the ReLU disjunction and the resulting large-scale optimization problem. This work investigates how pruning techniques can accelerate the solution of optimization problems with embedded neural networks, focusing on the mechanisms underlying the computational gains. We provide theoretical insights into how both unstructured (weight) and structured (node) pruning affect the ReLU big-M formulation, showing that pruning monotonically tightens preactivation bounds. We conduct comprehensive empirical studies across multiple network architectures using an illustrative test function and a realistic chemical process flowsheet optimization case study. Our results show that pruning achieves speedups of up to three to four orders of magnitude, with computational gains attributed to three key factors: (i) reduction in problem size, (ii) decrease in the number of integer variables, and (iii) tightening of big-M bounds. Weight pruning is particularly effective for deep, narrow networks, while node pruning performs better for shallow, wide or medium-sized networks. In the chemical engineering case study, pruning enabled convergence within seconds for problems that were otherwise intractable. We recommend adopting pruning as standard practice when developing neural network surrogates for optimization, especially for engineering applications requiring repeated optimization solves.