RbSyn: Type- and Effect-Guided Program Synthesis

TL;DR

RbSyn is a new synthesis tool for Ruby that efficiently generates effectful methods guided by types and effects, successfully handling side effects and passing test cases in under a minute for most benchmarks.

Contribution

Introduces RbSyn, a novel type- and effect-guided synthesis approach for effectful Ruby methods, formalized on a core language and scaled to real-world Ruby applications.

Findings

Successfully synthesizes correct solutions for all 19 benchmarks.

Most benchmarks are synthesized in under 9 seconds.

Type and effect guidance significantly improves synthesis performance.

Abstract

In recent years, researchers have explored component-based synthesis, which aims to automatically construct programs that operate by composing calls to existing APIs. However, prior work has not considered efficient synthesis of methods with side effects, e.g., web app methods that update a database. In this paper, we introduce RbSyn, a novel type- and effect-guided synthesis tool for Ruby. An RbSyn synthesis goal is specified as the type for the target method and a series of test cases it must pass. RbSyn works by recursively generating well-typed candidate method bodies whose write effects match the read effects of the test case assertions. After finding a set of candidates that separately satisfy each test, RbSyn synthesizes a solution that branches to execute the correct candidate code under the appropriate conditions. We formalize RbSyn on a core, object-oriented language $\lambda_{syn}$ and describe how the key ideas of the model are scaled-up in our implementation for Ruby. We evaluated RbSyn on 19 benchmarks, 12 of which come from popular, open-source Ruby apps. We found that RbSyn synthesizes correct solutions for all benchmarks, with 15 benchmarks synthesizing in under 9 seconds, while the slowest benchmark takes 83 seconds. Using observed reads to guide synthesize is effective: using type-guidance alone times out on 10 of 12 app benchmarks. We also found that using less precise effect annotations leads to worse synthesis performance. In summary, we believe type- and effect-guided synthesis is an important step forward in synthesis of effectful methods from test cases.