Choose, Don't Label: Multiple-Choice Query Synthesis for Program Disambiguation
Celeste Barnaby, Danny Ding, Osbert Bastani, Isil Dillig

TL;DR
This paper presents Socrates, an active learning system that uses multiple-choice queries to disambiguate high-level code specifications, improving reliability and efficiency over existing methods.
Contribution
It introduces a formalized multiple-choice query paradigm and an active learning algorithm for program disambiguation, implemented in the Socrates tool.
Findings
Socrates generates intuitive, easy-to-answer queries.
Socrates achieves more reliable program identification.
The system demonstrates efficient convergence across diverse domains.
Abstract
High-level specifications of code are inherently ambiguous, and prior systems have explored interactive techniques to help users clarify their intent and resolve such ambiguities. However, most existing approaches elicit supervision through labeled examples, which are often error-prone and may fail to capture user intent. This paper introduces a new active learning paradigm for program disambiguation based on multiple-choice queries. In this paradigm, the system presents a small set of high-level behaviors as multiple-choice options, and the user simply selects the intended one. Technically, each answer option corresponds to a Hoare triple that characterizes a cluster of semantically similar candidate programs. This formulation enables formal reasoning about the informativeness and interpretability of queries, and supports systematic construction of optimal queries. Building on this…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
