Program Synthesis Dialog Agents for Interactive Decision-Making
Matthew Toles, Nikhil Balwani, Rattandeep Singh, Valentina Giulia Sartori Rodriguez, Zhou Yu

TL;DR
This paper introduces ProADA, a program synthesis-based dialog agent for interactive decision-making in eligibility assessments, significantly improving accuracy over existing language models while maintaining dialog efficiency.
Contribution
The paper presents ProADA, a novel method that uses program synthesis for decision-making in dialog agents, addressing hallucination issues and enhancing performance on the BeNYfits benchmark.
Findings
ProADA achieves an F1 score of 55.6, outperforming GPT-4o's 35.7.
ProADA maintains similar dialog length while improving accuracy.
Current language models struggle with hallucinations in decision-making tasks.
Abstract
Many real-world eligibility problems, ranging from medical diagnosis to tax planning, can be mapped to decision problems expressed in natural language, wherein a model must make a binary choice based on user features. Large-scale domains such as legal codes or frequently updated funding opportunities render human annotation (e.g., web forms or decision trees) impractical, highlighting the need for agents that can automatically assist in decision-making. Since relevant information is often only known to the user, it is crucial that these agents ask the right questions. As agents determine when to terminate a conversation, they face a trade-off between accuracy and the number of questions asked, a key metric for both user experience and cost. To evaluate this task, we propose BeNYfits, a new benchmark for determining user eligibility for multiple overlapping social benefits opportunities…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsTopic Modeling · Multimodal Machine Learning Applications · AI in Service Interactions
