Multimodal Fused Learning for Solving the Generalized Traveling Salesman Problem in Robotic Task Planning

TL;DR

This paper introduces a multimodal fused learning framework that combines graph and image data to efficiently solve the generalized traveling salesman problem for robotic task planning, enabling real-time decision making.

Contribution

The paper presents a novel MMFL framework that integrates geometric and spatial features through multimodal fusion and adaptive scaling for improved GTSP solving in robotics.

Findings

Outperforms state-of-the-art methods on GTSP instances

Achieves real-time planning suitable for robotic applications

Validated on physical robots in real-world scenarios

Abstract

Effective and efficient task planning is essential for mobile robots, especially in applications like warehouse retrieval and environmental monitoring. These tasks often involve selecting one location from each of several target clusters, forming a Generalized Traveling Salesman Problem (GTSP) that remains challenging to solve both accurately and efficiently. To address this, we propose a Multimodal Fused Learning (MMFL) framework that leverages both graph and image-based representations to capture complementary aspects of the problem, and learns a policy capable of generating high-quality task planning schemes in real time. Specifically, we first introduce a coordinate-based image builder that transforms GTSP instances into spatially informative representations. We then design an adaptive resolution scaling strategy to enhance adaptability across different problem scales, and develop a multimodal fusion module with dedicated bottlenecks that enables effective integration of geometric and spatial features. Extensive experiments show that our MMFL approach significantly outperforms state-of-the-art methods across various GTSP instances while maintaining the computational efficiency required for real-time robotic applications. Physical robot tests further validate its practical effectiveness in real-world scenarios.