Analysis of AdvFusion: Adapter-based Multilingual Learning for Code Large Language Models

TL;DR

This paper evaluates AdvFusion, a PEFT approach for multilingual code models, across multiple tasks and models, revealing its strengths and limitations compared to other PEFT methods.

Contribution

It extends previous AdvFusion work to large language models and new tasks, providing a comprehensive analysis of its performance relative to other PEFT techniques.

Findings

AdvFusion outperforms AdapterFusion in code generation.

In commit message generation, AdapterFusion performs better than AdvFusion.

AdvFusion performs worse than AdapterFusion in code translation, especially as model size increases.

Abstract

Programming languages can benefit from one another by utilizing a language model for software engineering tasks. Full fine-tuning and Parameter Efficient Fine-Tuning (PEFT) of Code Language Models (Code-LMs) has been explored for multilingual knowledge transfer. AdapterFusion is a PEFT architecture that aims to enhance task performance by leveraging information from multiple programming languages, but primarily focuses on the target programming language. In our previous work, we proposed AdvFusion, a novel PEFT-based approach that effectively learns from other programming languages before adapting to the target task. Though previous experiments showed that AdvFusion outperformed AdapterFusion and LoRA, it was applied on pre-trained Code-LMs and was limited to only two tasks, code summarization and method name prediction. In this study, we expanded our work and investigated AdvFusion on Code Large Language Models (Code-LLMs), considering three new tasks: code generation, code translation, and commit message generation. We observed that different Code-LLMs/tasks exhibit different characteristics. In code generation, AdvFusion outperformed AdapterFusion but not other PEFT methods (LoRA, Compacter, and TaskAdapter). In commit message generation, AdapterFusion performed better than AdvFusion, and contrary to code generation, we found that the other PEFT methods do not have better performance. In code translation, AdvFusion performed worse than AdapterFusion overall, with the performance gap marginally widening as the model size increases. However, consistent with code generation, other PEFT methods showed better performance.