Readability-Robust Code Summarization via Meta Curriculum Learning

TL;DR

This paper evaluates the robustness of state-of-the-art code summarization models on poor-readability code and introduces RoFTCodeSum, a novel fine-tuning method that improves robustness and performance through curriculum and meta-learning.

Contribution

It presents RoFTCodeSum, a new fine-tuning approach combining curriculum and meta-learning to enhance code summarization robustness against poorly readable code.

Findings

State-of-the-art models drop in performance on poor-readability code.

Prompt engineering offers limited improvements.

RoFTCodeSum improves robustness and accuracy on obfuscated code.

Abstract

Code summarization has emerged as a fundamental technique in the field of program comprehension. While code language models have shown significant advancements, the current models and benchmarks are confined to high-readability code, which contains sufficient semantic cues such as function and variable names. In the real world, however, code is often poorly structured or obfuscated, significantly degrading model performance. In this paper, we first empirically evaluate the robustness of state-of-the-art language models on poor-readability code for the task of code summarization, focusing on (1) their effectiveness, (2) the impact of prompt engineering, and (3) the robustness of different variants. Experimental results reveal that state-of-the-art models-including GPT-4o and DeepSeek-V3 experience a substantial performance drop when faced with poorly readable code, and that prompt engineering and reasoning-enhanced models offer limited improvements. Motivated by these findings, we propose RoFTCodeSum, a novel fine-tuning method that enhances the robustness of code summarization against poorly readable code. RoFTCodeSum marries the concepts of curriculum learning and meta-learning: based on the original dataset for fine-tuning, it creates curricular training sets, e.g., obfuscating function names and identifiers from the code, respectively, that have progressive difficulty in code comprehension. In each training step, the approach meta-updates the gradients using these progressively challenging datasets, thereby optimizing both accuracy and readability robustness simultaneously. Experimental results demonstrate that RoFTCodeSum exhibits increased robustness against semantic perturbation while enhancing performance on the original code.