CodeQA: A Question Answering Dataset for Source Code Comprehension
Chenxiao Liu, Xiaojun Wan

TL;DR
CodeQA is a new dataset for source code comprehension that enables training and evaluation of question-answering models on Java and Python code snippets, facilitating advancements in code understanding.
Contribution
It introduces a large, natural question-answering dataset for source code, created through syntactic and semantic analysis of code comments, filling a gap in code comprehension research.
Findings
Neural models show promising results on CodeQA
The dataset contains over 189,000 question-answer pairs
Analysis highlights challenges and potential directions for code QA
Abstract
We propose CodeQA, a free-form question answering dataset for the purpose of source code comprehension: given a code snippet and a question, a textual answer is required to be generated. CodeQA contains a Java dataset with 119,778 question-answer pairs and a Python dataset with 70,085 question-answer pairs. To obtain natural and faithful questions and answers, we implement syntactic rules and semantic analysis to transform code comments into question-answer pairs. We present the construction process and conduct systematic analysis of our dataset. Experiment results achieved by several neural baselines on our dataset are shown and discussed. While research on question-answering and machine reading comprehension develops rapidly, few prior work has drawn attention to code question answering. This new dataset can serve as a useful research benchmark for source code comprehension.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsSoftware Engineering Research · Topic Modeling · Natural Language Processing Techniques
