Bridging Visualization and Optimization: Multimodal Large Language Models on Graph-Structured Combinatorial Optimization

TL;DR

This paper introduces a novel framework that transforms graph data into images to leverage multimodal large language models for solving complex graph-structured combinatorial problems with human-like spatial reasoning.

Contribution

It proposes converting graphs into images to utilize MLLMs for combinatorial optimization, demonstrating a new approach that mimics human spatial reasoning in machine problem-solving.

Findings

MLLMs show strong spatial reasoning abilities on graph tasks.

Transforming graphs into images improves problem-solving accuracy.

Simple search techniques combined with MLLMs are effective for complex graph challenges.

Abstract

Graph-structured combinatorial challenges are inherently difficult due to their nonlinear and intricate nature, often rendering traditional computational methods ineffective or expensive. However, these challenges can be more naturally tackled by humans through visual representations that harness our innate ability for spatial reasoning. In this study, we propose transforming graphs into images to preserve their higher-order structural features accurately, revolutionizing the representation used in solving graph-structured combinatorial tasks. This approach allows machines to emulate human-like processing in addressing complex combinatorial challenges. By combining the innovative paradigm powered by multimodal large language models (MLLMs) with simple search techniques, we aim to develop a novel and effective framework for tackling such problems. Our investigation into MLLMs spanned a variety of graph-based tasks, from combinatorial problems like influence maximization to sequential decision-making in network dismantling, as well as addressing six fundamental graph-related issues. Our findings demonstrate that MLLMs exhibit exceptional spatial intelligence and a distinctive capability for handling these problems, significantly advancing the potential for machines to comprehend and analyze graph-structured data with a depth and intuition akin to human cognition. These results also imply that integrating MLLMs with simple optimization strategies could form a novel and efficient approach for navigating graph-structured combinatorial challenges without complex derivations, computationally demanding training and fine-tuning.