Transducer Tuning: Efficient Model Adaptation for Software Tasks Using Code Property Graphs

TL;DR

This paper presents Transducer Tuning, a method that efficiently adapts large language models for software tasks by incorporating code structure via Code Property Graphs, significantly reducing training parameters while maintaining performance.

Contribution

Introduces Transducer Tuning, a modular approach that enriches code embeddings with structural information from CPGs, enabling efficient model adaptation without full fine-tuning.

Findings

Achieves competitive performance with up to 99% fewer trainable parameters.

Outperforms other tuning methods like LoRA and Prompt-Tuning in parameter efficiency.

Effective across multiple software engineering tasks such as code summarization and translation.

Abstract

Large language models have demonstrated promising performance across various software engineering tasks. While fine-tuning is a common practice to adapt these models for downstream tasks, it becomes challenging in resource-constrained environments due to increased memory requirements from growing trainable parameters in increasingly large language models. We introduce \approach, a technique to adapt large models for downstream code tasks using Code Property Graphs (CPGs). Our approach introduces a modular component called \transducer that enriches code embeddings with structural and dependency information from CPGs. The Transducer comprises two key components: Graph Vectorization Engine (GVE) and Attention-Based Fusion Layer (ABFL). GVE extracts CPGs from input source code and transforms them into graph feature vectors. ABFL then fuses those graphs feature vectors with initial code embeddings from a large language model. By optimizing these transducers for different downstream tasks, our approach enhances the models without the need to fine-tune them for specific tasks. We have evaluated \approach on three downstream tasks: code summarization, assert generation, and code translation. Our results demonstrate competitive performance compared to full parameter fine-tuning while reducing up to 99\% trainable parameters to save memory. \approach also remains competitive against other fine-tuning approaches (e.g., LoRA, Prompt-Tuning, Prefix-Tuning) while using only 1.5\%-80\% of their trainable parameters. Our findings show that integrating structural and dependency information through Transducer Tuning enables more efficient model adaptation, making it easier for users to adapt large models in resource-constrained settings.