An Effective Approach to Embedding Source Code by Combining Large Language and Sentence Embedding Models

TL;DR

This paper introduces a novel method for embedding source code by combining large language models and sentence embedding models, eliminating the need for task-specific training and improving robustness against errors.

Contribution

It proposes a new unsupervised approach that integrates LLMs with sentence embeddings for source code representation, outperforming existing methods without fine-tuning.

Findings

Outperforms state-of-the-art unsupervised methods

Effective across multiple programming languages

Reduces reliance on supervised training

Abstract

The advent of large language models (LLMs) has significantly advanced artificial intelligence (AI) in software engineering (SE), with source code embeddings playing a crucial role in tasks such as source code clone detection and source code clustering. However, existing methods for source code embedding, including those based on LLMs, often rely on costly supervised training or fine-tuning for domain adaptation. This paper proposes a novel approach to embedding source code by combining large language and sentence embedding models. This approach attempts to eliminate the need for task-specific training or fine-tuning and to effectively address the issue of erroneous information commonly found in LLM-generated outputs. To evaluate the performance of our proposed approach, we conducted a series of experiments on three datasets with different programming languages by considering various LLMs and sentence embedding models. The experimental results have demonstrated the effectiveness and superiority of our approach over the state-of-the-art unsupervised approaches, such as SourcererCC, Code2vec, InferCode, TransformCode, and LLM2Vec. Our findings highlight the potential of our approach to advance the field of SE by providing robust and efficient solutions for source code embedding tasks.