Active Learning for Neurosymbolic Program Synthesis

TL;DR

This paper introduces SmartLabel, an active learning method for neurosymbolic program synthesis that effectively handles neural mispredictions, achieving high accuracy with minimal user interaction.

Contribution

It proposes constrained conformal evaluation (CCE) to improve active learning in neurosymbolic synthesis, outperforming prior methods in accuracy and efficiency.

Findings

SmartLabel identifies the ground truth in 98% of benchmarks.

Requires under 5 rounds of interaction on average.

Outperforms prior techniques, which succeed in only 65% of benchmarks.

Abstract

The goal of active learning for program synthesis is to synthesize the desired program by asking targeted questions that minimize user interaction. While prior work has explored active learning in the purely symbolic setting, such techniques are inadequate for the increasingly popular paradigm of neurosymbolic program synthesis, where the synthesized program incorporates neural components. When applied to the neurosymbolic setting, such techniques can -- and, in practice, do -- return an unintended program due to mispredictions of neural components. This paper proposes a new active learning technique that can handle the unique challenges posed by neural network mispredictions. Our approach is based upon a new evaluation strategy called constrained conformal evaluation (CCE), which accounts for neural mispredictions while taking into account user-provided feedback. Our proposed method iteratively makes CCE more precise until all remaining programs are guaranteed to be observationally equivalent. We have implemented this method in a tool called SmartLabel and experimentally evaluated it on three neurosymbolic domains. Our results demonstrate that SmartLabel identifies the ground truth program for 98% of the benchmarks, requiring under 5 rounds of user interaction on average. In contrast, prior techniques for active learning are only able to converge to the ground truth program for at most 65% of the benchmarks.