Curiosity by Design: An LLM-based Coding Assistant Asking Clarification Questions

TL;DR

This paper presents a novel LLM-based coding assistant that improves code accuracy by asking clarification questions for ambiguous prompts, mimicking human review, and outperforming baseline models in usefulness and user satisfaction.

Contribution

The work introduces a system combining a query classifier and a fine-tuned LLM to generate clarification questions, enhancing coding assistance for ambiguous developer queries.

Findings

Fine-tuned LLM generates more useful clarification questions than zero-shot prompting.

Users prefer clarification questions from the proposed system over baseline models.

The system improves code accuracy and user satisfaction in coding assistance.

Abstract

Large Language Models (LLMs) are increasingly used as coding assistants. However, the ambiguity of the developer's prompt often leads to incorrect code generation, as current models struggle to infer user intent without extensive prompt engineering or external context. This work aims to build an LLM-based coding assistant that mimics the human code review process by asking clarification questions when faced with ambiguous or under-specified queries. Our end-to-end system includes (1) a query classifier trained to detect unclear programming-related queries and (2) a fine-tuned LLM that generates clarification questions. Our evaluation shows that the fine-tuned LLM outperforms standard zero-shot prompting in generating useful clarification questions. Furthermore, our user study indicates that users find the clarification questions generated by our model to outperform the baseline, demonstrating that our coding assistant produces more accurate and helpful code responses compared to baseline coding assistants.