On the Difficulty of Generalizing Reinforcement Learning Framework for Combinatorial Optimization

TL;DR

This paper investigates the generalization capabilities of reinforcement learning models for combinatorial optimization, specifically applying them to the quadratic assignment problem, and finds that current models may not generalize well.

Contribution

It provides an empirical evaluation of RL-based models on a classical quadratic assignment problem, highlighting limitations in their generalization across problem classes.

Findings

RL models struggle to generalize to quadratic assignment problems

Existing RL approaches perform well on specific problems like TSP

Generalization of RL models remains a significant challenge in combinatorial optimization

Abstract

Combinatorial optimization problems (COPs) on the graph with real-life applications are canonical challenges in Computer Science. The difficulty of finding quality labels for problem instances holds back leveraging supervised learning across combinatorial problems. Reinforcement learning (RL) algorithms have recently been adopted to solve this challenge automatically. The underlying principle of this approach is to deploy a graph neural network (GNN) for encoding both the local information of the nodes and the graph-structured data in order to capture the current state of the environment. Then, it is followed by the actor to learn the problem-specific heuristics on its own and make an informed decision at each state for finally reaching a good solution. Recent studies on this subject mainly focus on a family of combinatorial problems on the graph, such as the travel salesman problem, where the proposed model aims to find an ordering of vertices that optimizes a given objective function. We use the security-aware phone clone allocation in the cloud as a classical quadratic assignment problem (QAP) to investigate whether or not deep RL-based model is generally applicable to solve other classes of such hard problems. Extensive empirical evaluation shows that existing RL-based model may not generalize to QAP.