JuICe: A Large Scale Distantly Supervised Dataset for Open Domain Context-based Code Generation

TL;DR

JuICe is a large, curated dataset of 1.5 million examples designed for open-domain, context-based code generation in Jupyter notebooks, enabling improved training and evaluation of code generation models.

Contribution

The paper introduces JuICe, a large-scale, human-curated dataset for context-aware code generation, with new tasks and insights into model performance.

Findings

Context and distant supervision improve code generation.

Current models find JuICe challenging, indicating room for advancement.

JuICe enables training for API call sequence and full code cell generation.

Abstract

Interactive programming with interleaved code snippet cells and natural language markdown is recently gaining popularity in the form of Jupyter notebooks, which accelerate prototyping and collaboration. To study code generation conditioned on a long context history, we present JuICe, a corpus of 1.5 million examples with a curated test set of 3.7K instances based on online programming assignments. Compared with existing contextual code generation datasets, JuICe provides refined human-curated data, open-domain code, and an order of magnitude more training data. Using JuICe, we train models for two tasks: (1) generation of the API call sequence in a code cell, and (2) full code cell generation, both conditioned on the NL-Code history up to a particular code cell. Experiments using current baseline code generation models show that both context and distant supervision aid in generation, and that the dataset is challenging for current systems.