Refining Joint Text and Source Code Embeddings for Retrieval Task with Parameter-Efficient Fine-Tuning

TL;DR

This paper introduces a parameter-efficient fine-tuning framework using contrastive learning to enhance code-text retrieval with transformer models, significantly reducing computational costs while maintaining high performance.

Contribution

It proposes a novel PEFT-based fine-tuning approach with contrastive learning for bimodal representations, including extensive benchmarking of PEFT methods in this context.

Findings

Improves retrieval performance by tuning only 0.4% of parameters.

Demonstrates effectiveness of PEFT techniques on two datasets.

Provides comprehensive benchmarking of PEFT methods for code-text retrieval.

Abstract

The latest developments in Natural Language Processing (NLP) have demonstrated remarkable progress in a code-text retrieval problem. As the Transformer-based models used in this task continue to increase in size, the computational costs and time required for end-to-end fine-tuning become substantial. This poses a significant challenge for adapting and utilizing these models when computational resources are limited. Motivated by these concerns, we propose a fine-tuning framework that leverages Parameter-Efficient Fine-Tuning (PEFT) techniques. Moreover, we adopt contrastive learning objectives to improve the quality of bimodal representations learned by transformer models. Additionally, for PEFT methods we provide extensive benchmarking, the lack of which has been highlighted as a crucial problem in the literature. Based on the thorough experimentation with the CodeT5+ model conducted on two datasets, we demonstrate that the proposed fine-tuning framework has the potential to improve code-text retrieval performance by tuning only 0.4% parameters at most.