Can LLMs plan paths with extra hints from solvers?

TL;DR

This paper investigates enhancing large language models' planning abilities in robotic tasks by integrating solver-generated feedback, showing improvements on moderately difficult problems but limited success on harder ones.

Contribution

It introduces and evaluates four feedback strategies, including visual hints and fine-tuning, to improve LLMs' planning performance in robotic tasks.

Findings

Feedback improves performance on moderate problems

Hard problems remain challenging for LLMs

Different hinting strategies have varying effects

Abstract

Large Language Models (LLMs) have shown remarkable capabilities in natural language processing, mathematical problem solving, and tasks related to program synthesis. However, their effectiveness in long-term planning and higher-order reasoning has been noted to be limited and fragile. This paper explores an approach for enhancing LLM performance in solving a classical robotic planning task by integrating solver-generated feedback. We explore four different strategies for providing feedback, including visual feedback, we utilize fine-tuning, and we evaluate the performance of three different LLMs across a 10 standard and 100 more randomly generated planning problems. Our results suggest that the solver-generated feedback improves the LLM's ability to solve the moderately difficult problems, but the harder problems still remain out of reach. The study provides detailed analysis of the effects of the different hinting strategies and the different planning tendencies of the evaluated LLMs.