CodeQueries: A Dataset of Semantic Queries over Code

TL;DR

CodeQueries is a challenging dataset of semantic code questions over Python, designed to evaluate neural models' ability to understand code semantics at the file level through extractive question answering.

Contribution

The paper introduces CodeQueries, a novel dataset for semantic code question answering, supporting multi-hop reasoning and code span answers, based on static analysis tool queries.

Findings

Baseline neural models perform limitedly on CodeQueries.

CodeQueries challenges models to understand code semantics beyond simple yes/no questions.

The dataset includes both positive and negative examples with multi-hop reasoning.

Abstract

Developers often have questions about semantic aspects of code they are working on, e.g., "Is there a class whose parent classes declare a conflicting attribute?". Answering them requires understanding code semantics such as attributes and inheritance relation of classes. An answer to such a question should identify code spans constituting the answer (e.g., the declaration of the subclass) as well as supporting facts (e.g., the definitions of the conflicting attributes). The existing work on question-answering over code has considered yes/no questions or method-level context. We contribute a labeled dataset, called CodeQueries, of semantic queries over Python code. Compared to the existing datasets, in CodeQueries, the queries are about code semantics, the context is file level and the answers are code spans. We curate the dataset based on queries supported by a widely-used static analysis tool, CodeQL, and include both positive and negative examples, and queries requiring single-hop and multi-hop reasoning. To assess the value of our dataset, we evaluate baseline neural approaches. We study a large language model (GPT3.5-Turbo) in zero-shot and few-shot settings on a subset of CodeQueries. We also evaluate a BERT style model (CuBERT) with fine-tuning. We find that these models achieve limited success on CodeQueries. CodeQueries is thus a challenging dataset to test the ability of neural models, to understand code semantics, in the extractive question-answering setting.