Parameter-Efficient Multi-Task Fine-Tuning in Code-Related Tasks

TL;DR

This paper explores multi-task parameter-efficient fine-tuning of large language models for code tasks, demonstrating that QLoRA-based multi-task training can achieve competitive performance with reduced resource use and insights into model size effects.

Contribution

It is the first comprehensive study of multi-task QLoRA fine-tuning on code tasks, analyzing transfer learning effects and comparing with full fine-tuning and single-task approaches.

Findings

Multi-task QLoRA achieves competitive or better performance than single-task QLoRA.

Larger models balance correctness and quality more effectively.

Smaller models maintain functionality but have more quality issues.

Abstract

Large Language Models (LLMs) have proven highly effective in automating software engineering tasks, bridging natural language and code semantics to achieve notable results in code generation and summarization. However, their scale incurs substantial computational costs, making full fine-tuning impractical. Parameter-Efficient Fine-Tuning (PEFT) methods like QLoRA enable efficient specialization with lower resource demands. Recent studies show QLoRA-optimized Large Code Models (LCMs) perform strongly across diverse tasks, yet it remains unclear whether this effectiveness persists when a single model is QLoRA fine-tuned for multiple code-related tasks. The interaction between Multi-task fine-tuning and QLoRA optimization, and how transfer learning affects correctness and quality of generated artifacts, remains largely unexplored. We investigate Multi-task QLoRA fine-tuning across three representative tasks: code generation, translation, and summarization. We evaluate functional correctness through execution-based and similarity-based metrics, complemented by comprehensive code quality analysis--an aspect largely overlooked in prior work. Our findings show that Multi-task QLoRA effectively leverages transfer learning, achieving competitive or superior performance relative to both Single-task QLoRA and Multi-task full fine-tuning. Larger models demonstrate more consistent balance between correctness and quality, whereas smaller models preserve functionality but exhibit a higher incidence of quality-related issues.