SORREL: Suboptimal-Demonstration-Guided Reinforcement Learning for Learning to Branch

TL;DR

SORREL is a reinforcement learning approach that improves branching heuristics in MILP solvers by effectively leveraging suboptimal demonstrations, leading to better performance and training efficiency.

Contribution

It introduces a novel method that learns from suboptimal demonstrations using offline RL and self-imitation, reducing the need for high-quality demonstrations.

Findings

Outperforms previous methods in branching quality.

Achieves higher training efficiency.

Demonstrates effectiveness across various MILP problems.

Abstract

Mixed Integer Linear Program (MILP) solvers are mostly built upon a Branch-and-Bound (B\&B) algorithm, where the efficiency of traditional solvers heavily depends on hand-crafted heuristics for branching. The past few years have witnessed the increasing popularity of data-driven approaches to automatically learn these heuristics. However, the success of these methods is highly dependent on the availability of high-quality demonstrations, which requires either the development of near-optimal heuristics or a time-consuming sampling process. This paper averts this challenge by proposing Suboptimal-Demonstration-Guided Reinforcement Learning (SORREL) for learning to branch. SORREL selectively learns from suboptimal demonstrations based on value estimation. It utilizes suboptimal demonstrations through both offline reinforcement learning on the demonstrations generated by suboptimal heuristics and self-imitation learning on past good experiences sampled by itself. Our experiments demonstrate its advanced performance in both branching quality and training efficiency over previous methods for various MILPs.