Empirical Studies of Parameter Efficient Methods for Large Language Models of Code and Knowledge Transfer to R

TL;DR

This study evaluates PEFT methods like LoRA, Compacter, and IA^3 for large language models in code tasks, focusing on knowledge transfer to the under-explored language R, and compares their efficiency and effectiveness.

Contribution

It provides the first comprehensive evaluation of PEFT methods on large models for code in low-resource languages like R, highlighting LoRA's superior performance and resource efficiency.

Findings

LoRA outperforms other PEFT methods in all settings.

Compacter offers significant resource efficiency with minimal performance loss.

The number of trainable parameters impacts code accuracy more than PEFT architecture.

Abstract

Parameter Efficient Fine-Tuning (PEFT) methods are proposed as an alternative fine-tuning approach for Large Language Models (LLM) to minimize high training costs. While prior research demonstrates the effectiveness of PEFT methods in knowledge transfer using smaller language models, their application to larger LLMs, particularly in low-resource and unseen programming languages such as R, remains under-explored. In this work, we evaluate PEFT methods, LoRA, Compacter, and IA^3 on LLMs for code summarization and generation, with a particular emphasis on knowledge transfer to R as an unseen under-explored target language. Our experiments reveal that LoRA consistently outperforms Compacter and IA^3 in all settings, while Compacter offers significant resource efficiency with minimal performance trade-offs. Additionally, we find that the number of trainable parameters has a greater influence on the functional accuracy of the generated code than PEFT architecture. Our study can direct future research in developing code intelligent tasks for unseen languages including R, as well as the choice of PEFT methods for knowledge transfer, especially when balancing the computational cost and performance.