Robust Planning with LLM-Modulo Framework: Case Study in Travel Planning

TL;DR

This paper introduces the LLM-Modulo framework for enhancing large language models' planning capabilities, demonstrated through a travel planning case study, significantly improving performance on a benchmark compared to existing methods.

Contribution

The paper presents the LLM-Modulo framework, a novel approach that improves LLMs' planning performance, especially in travel planning, outperforming traditional reasoning enhancement techniques.

Findings

Performance improved by 4.6x over baseline for GPT4-Turbo.

Achieved 5% success rate with GPT3.5-Turbo, up from 0%.

Demonstrated useful roles of LLMs in planning pipelines.

Abstract

As the applicability of Large Language Models (LLMs) extends beyond traditional text processing tasks, there is a burgeoning interest in their potential to excel in planning and reasoning assignments, realms traditionally reserved for System 2 cognitive competencies. Despite their perceived versatility, the research community is still unraveling effective strategies to harness these models in such complex domains. The recent discourse introduced by the paper on LLM Modulo marks a significant stride, proposing a conceptual framework that enhances the integration of LLMs into diverse planning and reasoning activities. This workshop paper delves into the practical application of this framework within the domain of travel planning, presenting a specific instance of its implementation. We are using the Travel Planning benchmark by the OSU NLP group, a benchmark for evaluating the performance of LLMs in producing valid itineraries based on user queries presented in natural language. While popular methods of enhancing the reasoning abilities of LLMs such as Chain of Thought, ReAct, and Reflexion achieve a meager 0%, 0.6%, and 0% with GPT3.5-Turbo respectively, our operationalization of the LLM-Modulo framework for TravelPlanning domain provides a remarkable improvement, enhancing baseline performances by 4.6x for GPT4-Turbo and even more for older models like GPT3.5-Turbo from 0% to 5%. Furthermore, we highlight the other useful roles of LLMs in the planning pipeline, as suggested in LLM-Modulo, which can be reliably operationalized such as extraction of useful critics and reformulator for critics.