The Usability of Pragmatic Communication in Regular Expression Synthesis

TL;DR

This paper investigates how pragmatic communication improves the usability of program synthesis for regular expressions, enabling users to specify targets more effectively with fewer examples, especially when negative examples are unavailable.

Contribution

It adapts pragmatic reasoning to regex synthesis and provides a rigorous user study demonstrating improved communication success and efficiency.

Findings

Pragmatic synthesizer achieves 95% success with positive examples alone.

Users communicate more efficiently, reducing examples by 57%.

Pragmatic approach outperforms non-pragmatic methods in user studies.

Abstract

Programming-by-example (PBE) systems aim to alleviate the burden of programming. However, user-specified examples are often ambiguous, leaving multiple programs to satisfy the specification. Consequently, in most prior work, users have had to provide additional examples, particularly negative ones, to further constrain the search over compatible programs. Recent work resolves additional ambiguity by modeling program synthesis tasks as pragmatic communication, showing promising results on a graphics domain using a rudimentary user-study. We adapt pragmatic reasoning to a sub-domain of regular expressions and rigorously study its usability as a means of communication both with and without the ability to provide negative examples. Our user study (N=30) demonstrates that, with a pragmatic synthesizer, end-users can more successfully communicate a target regex using positive examples alone (95%) compared to using a non-pragmatic synthesizer (51%). Further, users can communicate more efficiently (57% fewer examples) with a pragmatic synthesizer compared to a non-pragmatic one.