When Fine-Tuning LLMs Meets Data Privacy: An Empirical Study of Federated Learning in LLM-Based Program Repair

TL;DR

This paper explores using federated learning to fine-tune large language models for automated program repair while preserving data privacy across private code repositories.

Contribution

It demonstrates that federated fine-tuning improves program repair capabilities and shows robustness to data heterogeneity in real-world industry scenarios.

Findings

Federated fine-tuning enhances LLM-based program repair.

Heterogeneous code data has negligible impact on fine-tuning effectiveness.

Different federated algorithms have unique strengths depending on the LLM used.

Abstract

Software systems have been evolving rapidly and inevitably introducing bugs at an increasing rate, leading to significant losses in resources consumed by software maintenance. Recently, large language models (LLMs) have demonstrated remarkable potential in enhancing software development and maintenance practices, particularly in automated program repair (APR) with improved accuracy and efficiency of bug fixing. However, LLM-based APR heavily relies on high-quality code repositories. A larger portion of existing code repositories are for private use and proprietary assets from various industries, reflecting more diversity and nuances in the data since real-world industries often have more extensive software development practices, which cannot be covered by merely public datasets. Therefore, utilizing private datasets shows significant potential in enhancing software development and maintenance. However, obtaining such data from various industries is hindered by data privacy concerns, as companies are reluctant to share their codebases. To address the gap, we investigate the use of federated learning as a privacy-preserving approach that enables private entities to fine-tune LLMs on proprietary and decentralized data, facilitating the collaboration between clients to fully utilize their data to help enhance software development and maintenance. Our evaluation reveals that federated fine-tuning can effectively enhance program repair capabilities. Notably, the impact of heterogeneous code on LLM fine-tuning is negligible, indicating that real-world industries can benefit from collaborative development regardless of diverse data distributions. Furthermore, each type of federated algorithm exhibits unique strengths across different LLMs, suggesting that fine-tuning for program repair can be enhanced by tailoring the optimization process to specific characteristics of different LLMs.