AdvFusion: Adapter-based Knowledge Transfer for Code Summarization on Code Language Models

TL;DR

AdvFusion introduces a parameter-efficient fine-tuning method that leverages multilingual knowledge transfer to improve code summarization and method name prediction in code language models, outperforming existing PEFT approaches.

Contribution

The paper proposes AdvFusion, a novel PEFT approach that learns from multiple languages before adapting to specific tasks, enhancing multilingual knowledge transfer in code models.

Findings

AdvFusion outperforms AdapterFusion by up to 1.7 points.

AdvFusion surpasses LoRA with gains of 1.99, 1.26, and 2.16 on Ruby, JavaScript, and Go.

The approach improves code summarization and method name prediction accuracy.

Abstract

Programming languages can benefit from one another by utilizing a pre-trained model for software engineering tasks such as code summarization and method name prediction. While full fine-tuning of Code Language Models (Code-LMs) has been explored for multilingual knowledge transfer, research on Parameter Efficient Fine-Tuning (PEFT) for this purpose is limited. AdapterFusion, a PEFT architecture, aims to enhance task performance by leveraging information from multiple languages but primarily focuses on the target language. To address this, we propose AdvFusion, a novel PEFT-based approach that effectively learns from other languages before adapting to the target task. Evaluated on code summarization and method name prediction, AdvFusion outperforms AdapterFusion by up to 1.7 points and surpasses LoRA with gains of 1.99, 1.26, and 2.16 for Ruby, JavaScript, and Go, respectively. We open-source our scripts for replication purposes.