Text Classification for Task-based Source Code Related Questions

TL;DR

This paper presents a deep learning approach combining Seq2Seq and binary classification to improve the matching of natural language task descriptions with Python code snippets, demonstrating the effectiveness of hidden state embeddings.

Contribution

It introduces a novel model that leverages hidden state embeddings from Seq2Seq for better intent-code matching in code generation tasks.

Findings

Hidden state embeddings outperform standard embeddings.

Using Seq2Seq hidden states improves intent-code matching accuracy.

Pre-trained code embeddings are less context-aware than Seq2Seq embeddings.

Abstract

There is a key demand to automatically generate code for small tasks for developers. Websites such as StackOverflow provide a simplistic way by offering solutions in small snippets which provide a complete answer to whatever task question the developer wants to code. Natural Language Processing and particularly Question-Answering Systems are very helpful in resolving and working on these tasks. In this paper, we develop a two-fold deep learning model: Seq2Seq and a binary classifier that takes in the intent (which is in natural language) and code snippets in Python. We train both the intent and the code utterances in the Seq2Seq model, where we decided to compare the effect of the hidden layer embedding from the encoder for representing the intent and similarly, using the decoder's hidden layer embeddings for the code sequence. Then we combine both these embeddings and then train a simple binary neural network classifier model for predicting if the intent is correctly answered by the predicted code sequence from the seq2seq model. We find that the hidden state layer's embeddings perform slightly better than regular standard embeddings from a constructed vocabulary. We experimented with our tests on the CoNaLa dataset in addition to the StaQC database consisting of simple task-code snippet-based pairs. We empirically establish that using additional pre-trained embeddings for code snippets in Python is less context-based in comparison to using hidden state context vectors from seq2seq models.