I Need Help! Evaluating LLM's Ability to Ask for Users' Support: A Case Study on Text-to-SQL Generation

TL;DR

This paper investigates how large language models can proactively seek user support in text-to-SQL tasks, proposing metrics to evaluate when and how models should ask for help to balance performance and user burden.

Contribution

It introduces new metrics for assessing support-seeking behavior in LLMs and analyzes their ability to determine when to request user assistance under different information conditions.

Findings

Many LLMs struggle to recognize their need for support without external feedback.

External signals significantly improve models' ability to seek help appropriately.

Insights provided for future development of support-seeking strategies in LLMs.

Abstract

This study explores the proactive ability of LLMs to seek user support. We propose metrics to evaluate the trade-off between performance improvements and user burden, and investigate whether LLMs can determine when to request help under varying information availability. Our experiments show that without external feedback, many LLMs struggle to recognize their need for user support. The findings highlight the importance of external signals and provide insights for future research on improving support-seeking strategies. Source code: https://github.com/appier-research/i-need-help