Reinforcement Learning Methods for Neighborhood Selection in Local Search

TL;DR

This paper evaluates reinforcement learning strategies for neighborhood selection in local search algorithms across various combinatorial problems, highlighting their potential and practical limitations.

Contribution

It provides a comprehensive comparison of RL-based neighborhood selection methods, emphasizing the importance of reward design and analyzing their performance and computational costs.

Findings

$oldsymbol{ ext{ extit{ extbf{epsilon}}-greedy}}$ performs consistently well.

Deep RL methods are computationally intensive and less practical.

Reward function design critically affects learning stability.

Abstract

Reinforcement learning has recently gained traction as a means to improve combinatorial optimization methods, yet its effectiveness within local search metaheuristics specifically remains comparatively underexamined. In this study, we evaluate a range of reinforcement learning-based neighborhood selection strategies -- multi-armed bandits (upper confidence bound, $\epsilon$-greedy) and deep reinforcement learning methods (proximal policy optimization, double deep $Q$-network) -- and compare them against multiple baselines across three different problems: the traveling salesman problem, the pickup and delivery problem with time windows, and the car sequencing problem. We show how search-specific characteristics, particularly large variations in cost due to constraint violation penalties, necessitate carefully designed reward functions to provide stable and informative learning signals. Our extensive experiments reveal that algorithm performance varies substantially across problems, although that $\epsilon$-greedy consistently ranks among the best performers. In contrast, the computational overhead of deep reinforcement learning approaches only makes them competitive with a substantially longer runtime. These findings highlight both the promise and the practical limitations of deep reinforcement learning in local search.