Towards Neural Functional Program Evaluation

TL;DR

This paper investigates the use of transformer-based language models, specifically T5, for evaluating simple functional programs, demonstrating high accuracy and exploring generalization across different program features.

Contribution

It introduces a new program generation method controlling syntactic sugar and evaluates neural program evaluation's effectiveness and generalization capabilities.

Findings

T5 achieves over 90% exact match scores on in-distribution tests.

Pretrained T5 weights outperform random initialization.

Models generalize well across type, function composition, and reduction steps.

Abstract

This paper explores the capabilities of current transformer-based language models for program evaluation of simple functional programming languages. We introduce a new program generation mechanism that allows control over syntactic sugar for semantically equivalent programs. T5 experiments reveal that neural functional program evaluation performs surprisingly well, achieving high 90% exact program match scores for most in-distribution and out-of-distribution tests. Using pretrained T5 weights has significant advantages over random initialization. We present and evaluate on three datasets to study generalization abilities that are specific to functional programs based on: type, function composition, and reduction steps. Code and data are publicly available at https://github.com/ElementAI/neural-interpreters.