Routing Arena: A Benchmark Suite for Neural Routing Solvers

TL;DR

The paper introduces Routing Arena, a benchmark suite for neural routing solvers that standardizes evaluation protocols, compares neural and traditional methods, and proposes a new metric to assess runtime efficiency.

Contribution

It provides a comprehensive benchmark suite with consistent evaluation protocols and a novel metric, WRAP, for neural routing solvers, addressing current evaluation flaws.

Findings

State-of-the-art OR solvers excel in solution quality and runtime.

Neural approaches show advantages in certain aspects, encouraging a paradigm shift.

Routing Arena enables fair comparison between neural and traditional routing methods.

Abstract

Neural Combinatorial Optimization has been researched actively in the last eight years. Even though many of the proposed Machine Learning based approaches are compared on the same datasets, the evaluation protocol exhibits essential flaws and the selection of baselines often neglects State-of-the-Art Operations Research approaches. To improve on both of these shortcomings, we propose the Routing Arena, a benchmark suite for Routing Problems that provides a seamless integration of consistent evaluation and the provision of baselines and benchmarks prevalent in the Machine Learning- and Operations Research field. The proposed evaluation protocol considers the two most important evaluation cases for different applications: First, the solution quality for an a priori fixed time budget and secondly the anytime performance of the respective methods. By setting the solution trajectory in perspective to a Best Known Solution and a Base Solver's solutions trajectory, we furthermore propose the Weighted Relative Average Performance (WRAP), a novel evaluation metric that quantifies the often claimed runtime efficiency of Neural Routing Solvers. A comprehensive first experimental evaluation demonstrates that the most recent Operations Research solvers generate state-of-the-art results in terms of solution quality and runtime efficiency when it comes to the vehicle routing problem. Nevertheless, some findings highlight the advantages of neural approaches and motivate a shift in how neural solvers should be conceptualized.