Optimization over Trained (and Sparse) Neural Networks: A Surrogate within a Surrogate

TL;DR

This paper explores how pruning large, pre-trained neural networks can create more tractable surrogate models for optimization problems, often improving solution quality even without finetuning.

Contribution

It introduces a method to use pruned, sparse neural networks as surrogates in optimization, bypassing the need for retraining to achieve better tractability and solutions.

Findings

Pruned networks can outperform fine-tuned ones as surrogates.

Skipping finetuning can lead to better solutions within time limits.

Pruning enhances tractability of neural network-based optimization models.

Abstract

In constraint learning, we use a neural network as a surrogate for part of the constraints or of the objective function of an optimization model. However, the tractability of the resulting model is heavily influenced by the size of the neural network used as a surrogate. One way to obtain a more tractable surrogate is by pruning the neural network first. In this work, we consider how to approach the setting in which the neural network is actually a given: how can we solve an optimization model embedding a large and predetermined neural network? We propose surrogating the neural network itself by pruning it, which leads to a sparse and more tractable optimization model, for which we hope to still obtain good solutions with respect to the original neural network. For network verification and function maximization models, that indeed leads to better solutions within a time limit, especially -- and surprisingly -- if we skip the standard retraining step known as finetuning. Hence, a pruned network with worse inference for lack of finetuning can be a better surrogate.