RobustFill: Neural Program Learning under Noisy I/O

TL;DR

This paper compares neural program synthesis and induction approaches on a real-world task, demonstrating that neural models can achieve high accuracy and robustness to noise, outperforming rule-based methods.

Contribution

It provides the first large-scale comparison of neural program synthesis and induction, highlighting their strengths and limitations in real-world noisy data scenarios.

Findings

Neural synthesis model achieves 92% accuracy on real-world data.

Neural models are robust to noise such as typos.

Rule-based systems fail under noisy conditions.

Abstract

The problem of automatically generating a computer program from some specification has been studied since the early days of AI. Recently, two competing approaches for automatic program learning have received significant attention: (1) neural program synthesis, where a neural network is conditioned on input/output (I/O) examples and learns to generate a program, and (2) neural program induction, where a neural network generates new outputs directly using a latent program representation. Here, for the first time, we directly compare both approaches on a large-scale, real-world learning task. We additionally contrast to rule-based program synthesis, which uses hand-crafted semantics to guide the program generation. Our neural models use a modified attention RNN to allow encoding of variable-sized sets of I/O pairs. Our best synthesis model achieves 92% accuracy on a real-world test set, compared to the 34% accuracy of the previous best neural synthesis approach. The synthesis model also outperforms a comparable induction model on this task, but we more importantly demonstrate that the strength of each approach is highly dependent on the evaluation metric and end-user application. Finally, we show that we can train our neural models to remain very robust to the type of noise expected in real-world data (e.g., typos), while a highly-engineered rule-based system fails entirely.