Task-oriented Prompt Enhancement via Script Generation

TL;DR

TITAN introduces a zero-shot, script-generation approach that enhances large language models' ability to handle task-oriented prompts without manual templates, significantly improving performance across diverse tasks.

Contribution

This work presents TITAN, a novel zero-shot script generation method that eliminates manual prompt engineering and improves LLMs' task performance using chain-of-thought and step-back prompting techniques.

Findings

TITAN outperforms state-of-the-art zero-shot methods by 7.6% with GPT-3.5.

TITAN achieves state-of-the-art results in 8 out of 11 tasks without human annotation.

TITAN's approach is effective across diverse task types, demonstrating broad applicability.

Abstract

Large Language Models (LLMs) have demonstrated remarkable abilities across various tasks, leveraging advanced reasoning. Yet, they struggle with task-oriented prompts due to a lack of specific prior knowledge of the task answers. The current state-of-the-art approach, PAL, utilizes code generation to address this issue. However, PAL depends on manually crafted prompt templates and examples while still producing inaccurate results. In this work, we present TITAN-a novel strategy designed to enhance LLMs' performance on task-oriented prompts. TITAN achieves this by generating scripts using a universal approach and zero-shot learning. Unlike existing methods, TITAN eliminates the need for detailed task-specific instructions and extensive manual efforts. TITAN enhances LLMs' performance on various tasks by utilizing their analytical and code-generation capabilities in a streamlined process. TITAN employs two key techniques: (1) step-back prompting to extract the task's input specifications and (2) chain-of-thought prompting to identify required procedural steps. This information is used to improve the LLMs' code-generation process. TITAN further refines the generated script through post-processing and the script is executed to retrieve the final answer. Our comprehensive evaluation demonstrates TITAN's effectiveness in a diverse set of tasks. On average, TITAN outperforms the state-of-the-art zero-shot approach by 7.6% and 3.9% when paired with GPT-3.5 and GPT-4. Overall, without human annotation, TITAN achieves state-of-the-art performance in 8 out of 11 cases while only marginally losing to few-shot approaches (which needed human intervention) on three occasions by small margins. This work represents a significant advancement in addressing task-oriented prompts, offering a novel solution for effectively utilizing LLMs in everyday life tasks.