A Code Comprehension Benchmark for Large Language Models for Code

TL;DR

This paper introduces a benchmark to evaluate and improve large language models' understanding of code semantics through fine-tuning, demonstrating significant performance gains on code comprehension tasks.

Contribution

It proposes a new benchmark for code comprehension and shows that fine-tuning models on relevant tasks enhances their semantic understanding capabilities.

Findings

Fine-tuning improves model accuracy on code comprehension tasks.

QWQ-32B model accuracy increased from 70% to 83.47%.

DPO-fine-tuned Codestral-22B achieved 87.66% micro-accuracy.

Abstract

Large Language Models have shown impressive capabilities in coding tasks like code generation and code completion, as they have been trained on a large amount of code data. Also, since one of the core pretraining objectives is Next Token Prediction, these models tends to learn surface-level syntactic patterns in code. However, this does not guarantee code comprehension ability i.e. the ability to capture the semantics of the code. In our opinion, this is the reason why these models often underperform on tasks that require deeper semantic understanding, such as code debugging and code optimization. To address this, we propose fine-tuning these models specifically for code comprehension tasks using large-scale datasets, enabling them to develop a more robust understanding of code semantics. We evaluate three code models of varying sizes on a suite of code comprehension tasks designed to assess semantic understanding beyond surface-level syntactic pattern matching. In particular, we analyze performance on the Subjectivity Grading Task and observe that model performance improves after fine-tuning on relevant downstream tasks. The most significant improvement is seen in the QWQ-32B model, where accuracy increases from 70% to 83.47%. A similar or explainable trend is observed across other models, clearly indicating an enhancement in code comprehension ability. Among the models studied, the DPO-fine-tuned Codestral-22B achieves the highest micro-accuracy of 87.66% on the Subjectivity Grading Task.