Behavior and Representation in Large Language Models for Combinatorial Optimization: From Feature Extraction to Algorithm Selection

TL;DR

This paper explores how large language models understand and represent combinatorial optimization problems, revealing their potential to support feature extraction and algorithm selection through internal representations.

Contribution

It introduces a probing framework to analyze LLMs' internal encoding of problem features and demonstrates their capability to predict optimal solvers for various benchmark problems.

Findings

LLMs can moderately recover problem features via direct querying.

Hidden-layer representations of LLMs are as effective as traditional features for prediction.

LLMs encode meaningful structural information relevant to optimization tasks.

Abstract

Recent advances in Large Language Models (LLMs) have opened new perspectives for automation in optimization. While several studies have explored how LLMs can generate or solve optimization models, far less is understood about what these models actually learn regarding problem structure or algorithmic behavior. This study investigates how LLMs internally represent combinatorial optimization problems and whether such representations can support downstream decision tasks. We adopt a twofold methodology combining direct querying, which assesses LLM capacity to explicitly extract instance features, with probing analyses that examine whether such information is implicitly encoded within their hidden layers. The probing framework is further extended to a per-instance algorithm selection task, evaluating whether LLM-derived representations can predict the best-performing solver. Experiments span four benchmark problems and three instance representations. Results show that LLMs exhibit moderate ability to recover feature information from problem instances, either through direct querying or probing. Notably, the predictive power of LLM hidden-layer representations proves comparable to that achieved through traditional feature extraction, suggesting that LLMs capture meaningful structural information relevant to optimization performance.