Mis-prompt: Benchmarking Large Language Models for Proactive Error Handling

TL;DR

This paper introduces Mis-prompt, a benchmark for evaluating large language models' ability to handle errors proactively without explicit instructions, revealing current limitations and potential improvements through supervised fine-tuning.

Contribution

It presents a new benchmark, evaluation tasks, and dataset for proactive error handling in LLMs, addressing a gap in current error management research.

Findings

Current LLMs perform poorly on proactive error handling.

Supervised fine-tuning improves LLMs' error handling capabilities.

The dataset will be publicly available for further research.

Abstract

Large language models (LLMs) have demonstrated significant advancements in error handling. Current error-handling works are performed in a passive manner, with explicit error-handling instructions. However, in real-world scenarios, explicit error-handling instructions are usually unavailable. In this paper, our work identifies this challenge as how to conduct proactive error handling without explicit error handling instructions. To promote further research, this work introduces a new benchmark, termed Mis-prompt, consisting of four evaluation tasks, an error category taxonomy, and a new evaluation dataset. Furthermore, this work analyzes current LLMs' performance on the benchmark, and the experimental results reveal that current LLMs show poor performance on proactive error handling, and SFT on error handling instances improves LLMs' proactive error handling capabilities. The dataset will be publicly available.