Smart Language Agents in Real-World Planning

TL;DR

This paper enhances travel planning with Large Language Models by introducing a semi-automated prompt refinement method that significantly improves planning accuracy through human-in-the-loop feedback.

Contribution

It proposes a semi-automated prompt generation framework combining LLM automation and human input to boost travel planning performance.

Findings

Human-in-the-loop improves LLM performance by 139%

Single iteration of prompt refinement yields substantial gains

Automated prompts alone have limitations in travel planning tasks

Abstract

Comprehensive planning agents have been a long term goal in the field of artificial intelligence. Recent innovations in Natural Language Processing have yielded success through the advent of Large Language Models (LLMs). We seek to improve the travel-planning capability of such LLMs by extending upon the work of the previous paper TravelPlanner. Our objective is to explore a new method of using LLMs to improve the travel planning experience. We focus specifically on the "sole-planning" mode of travel planning; that is, the agent is given necessary reference information, and its goal is to create a comprehensive plan from the reference information. While this does not simulate the real-world we feel that an optimization of the sole-planning capability of a travel planning agent will still be able to enhance the overall user experience. We propose a semi-automated prompt generation framework which combines the LLM-automated prompt and "human-in-the-loop" to iteratively refine the prompt to improve the LLM performance. Our result shows that LLM automated prompt has its limitations and "human-in-the-loop" greatly improves the performance by $139\%$ with one single iteration.