Disentangling Ambiguity from Instability in Large Language Models: A Clinical Text-to-SQL Case Study

TL;DR

This paper introduces CLUES, a framework for clinical Text-to-SQL that distinguishes input ambiguity from model instability, enabling targeted interventions and improved failure prediction.

Contribution

CLUES models Text-to-SQL as a two-stage process, decomposes semantic uncertainty into ambiguity and instability scores, and enhances failure prediction in clinical NLP tasks.

Findings

CLUES improves failure prediction over Kernel Language Entropy.

It provides a diagnostic decomposition of uncertainty unavailable from a single score.

The high-ambiguity/high-instability regime contains 51% of errors while covering 25% of queries.

Abstract

Deploying large language models for clinical Text-to-SQL requires distinguishing two qualitatively different causes of output diversity: (i) input ambiguity that should trigger clarification, and (ii) model instability that should trigger human review. We propose CLUES, a framework that models Text-to-SQL as a two-stage process (interpretations --> answers) and decomposes semantic uncertainty into an ambiguity score and an instability score. The instability score is computed via the Schur complement of a bipartite semantic graph matrix. Across AmbigQA/SituatedQA (gold interpretations) and a clinical Text-to-SQL benchmark (known interpretations), CLUES improves failure prediction over state-of-the-art Kernel Language Entropy. In deployment settings, it remains competitive while providing a diagnostic decomposition unavailable from a single score. The resulting uncertainty regimes map to targeted interventions - query refinement for ambiguity, model improvement for instability. The high-ambiguity/high-instability regime contains 51% of errors while covering 25% of queries, enabling efficient triage.