Exploring ReAct Prompting for Task-Oriented Dialogue: Insights and Shortcomings

TL;DR

This paper investigates the application of ReAct prompting strategy to task-oriented dialogue with large language models, revealing its strengths in human satisfaction but limitations in success rate compared to state-of-the-art methods.

Contribution

It is the first to evaluate ReAct prompting in task-oriented dialogue both in simulation and with real users, highlighting its advantages and shortcomings.

Findings

ReAct-LLMs underperform in success rate in simulation.

Humans report higher satisfaction with ReAct-LLMs.

ReAct-LLMs produce more natural and confident responses.

Abstract

Large language models (LLMs) gained immense popularity due to their impressive capabilities in unstructured conversations. Empowering LLMs with advanced prompting strategies such as reasoning and acting (ReAct) (Yao et al., 2022) has shown promise in solving complex tasks traditionally requiring reinforcement learning. In this work, we apply the ReAct strategy to guide LLMs performing task-oriented dialogue (TOD). We evaluate ReAct-based LLMs (ReAct-LLMs) both in simulation and with real users. While ReAct-LLMs severely underperform state-of-the-art approaches on success rate in simulation, this difference becomes less pronounced in human evaluation. Moreover, compared to the baseline, humans report higher subjective satisfaction with ReAct-LLM despite its lower success rate, most likely thanks to its natural and confidently phrased responses.