BenLOC: A Benchmark for Learning to Configure MIP Optimizers

TL;DR

BenLOC is a standardized benchmark and toolkit designed to fairly evaluate machine learning methods for configuring MIP solvers, addressing previous issues of data leakage and inconsistent evaluation.

Contribution

It introduces a comprehensive benchmark and open-source toolkit for unbiased evaluation of MIP optimizer configuration methods, standardizing datasets, features, and experimental setups.

Findings

Classical ML models with handcrafted features perform competitively.

Dataset choice significantly impacts evaluation outcomes.

BenLOC provides unbiased, comprehensive benchmarking results.

Abstract

The automatic configuration of Mixed-Integer Programming (MIP) optimizers has become increasingly critical as the large number of configurations can significantly affect solver performance. Yet the lack of standardized evaluation frameworks has led to data leakage and over-optimistic claims, as prior studies often rely on homogeneous datasets and inconsistent experimental setups. To promote a fair evaluation process, we present BenLOC, a comprehensive benchmark and open-source toolkit, which not only offers an end-to-end pipeline for learning instance-wise MIP optimizer configurations, but also standardizes dataset selection, train-test splits, feature engineering and baseline choice for unbiased and comprehensive evaluations. Leveraging this framework, we conduct an empirical analysis on five well-established MIP datasets and compare classical machine learning models with handcrafted features against state-of-the-art deep-learning techniques. The results demonstrate the importance of datasets, features and baseline criteria proposed by BenLOC and the effectiveness of BenLOC in providing unbiased and comprehensive evaluations.