Can humans teach machines to code?

TL;DR

This paper investigates whether humans can effectively teach machines to code by providing input-output examples, revealing that non-experts often do not supply enough information for accurate program synthesis.

Contribution

The study evaluates the ability of humans, both experts and non-experts, to teach program synthesis systems through examples, highlighting limitations of non-expert input.

Findings

Non-experts generally do not provide enough examples for accurate learning

Expert-provided examples lead to better generalization

Random samples perform poorly in teaching programs

Abstract

The goal of inductive program synthesis is for a machine to automatically generate a program from user-supplied examples. A key underlying assumption is that humans can provide sufficient examples to teach a concept to a machine. To evaluate the validity of this assumption, we conduct a study where human participants provide examples for six programming concepts, such as finding the maximum element of a list. We evaluate the generalisation performance of five program synthesis systems trained on input-output examples (i) from non-expert humans, (ii) from a human expert, and (iii) randomly sampled. Our results suggest that non-experts typically do not provide sufficient examples for a program synthesis system to learn an accurate program.