Auto-Prompting with Retrieval Guidance for Frame Detection in Logistics

TL;DR

This paper introduces a prompt optimization framework combining retrieval, few-shot learning, and automatic chain-of-thought synthesis to improve frame detection in logistics texts, achieving significant accuracy gains without fine-tuning.

Contribution

It presents a novel prompt optimization pipeline with an LLM-based agent that iteratively refines prompts using retrieval and self-evaluation, enhancing domain-specific NLP tasks.

Findings

Up to 15% improvement in inference accuracy with optimized prompts.

Effective across multiple LLMs including GPT-4o, Qwen 2.5, and LLaMA 3.1.

Demonstrates structured prompt optimization as a scalable alternative to fine-tuning.

Abstract

Prompt engineering plays a critical role in adapting large language models (LLMs) to complex reasoning and labeling tasks without the need for extensive fine-tuning. In this paper, we propose a novel prompt optimization pipeline for frame detection in logistics texts, combining retrieval-augmented generation (RAG), few-shot prompting, chain-of-thought (CoT) reasoning, and automatic CoT synthesis (Auto-CoT) to generate highly effective task-specific prompts. Central to our approach is an LLM-based prompt optimizer agent that iteratively refines the prompts using retrieved examples, performance feedback, and internal self-evaluation. Our framework is evaluated on a real-world logistics text annotation task, where reasoning accuracy and labeling efficiency are critical. Experimental results show that the optimized prompts - particularly those enhanced via Auto-CoT and RAG - improve real-world inference accuracy by up to 15% compared to baseline zero-shot or static prompts. The system demonstrates consistent improvements across multiple LLMs, including GPT-4o, Qwen 2.5 (72B), and LLaMA 3.1 (70B), validating its generalizability and practical value. These findings suggest that structured prompt optimization is a viable alternative to full fine-tuning, offering scalable solutions for deploying LLMs in domain-specific NLP applications such as logistics.