Selective Shot Learning for Code Explanation

TL;DR

This paper introduces SSL_ner, a novel selective shot learning method using entity information for code explanation, and provides the first systematic benchmarking of open-source Code-LLMs with various few-shot selection techniques.

Contribution

It proposes SSL_ner, a new method for few-shot example selection in code explanation, and conducts the first comprehensive benchmarking of open-source Code-LLMs and SSL approaches.

Findings

SSL_ner outperforms existing methods on two datasets.

Benchmarking reveals insights into open-source Code-LLMs performance.

Entity-based selection improves few-shot learning effectiveness.

Abstract

Code explanation plays a crucial role in the software engineering domain, aiding developers in grasping code functionality efficiently. Recent work shows that the performance of LLMs for code explanation improves in a few-shot setting, especially when the few-shot examples are selected intelligently. State-of-the-art approaches for such Selective Shot Learning (SSL) include token-based and embedding-based methods. However, these SSL approaches have been evaluated on proprietary LLMs, without much exploration on open-source Code-LLMs. Additionally, these methods lack consideration for programming language syntax. To bridge these gaps, we present a comparative study and propose a novel SSL method (SSL_ner) that utilizes entity information for few-shot example selection. We present several insights and show the effectiveness of SSL_ner approach over state-of-the-art methods across two datasets. To the best of our knowledge, this is the first systematic benchmarking of open-source Code-LLMs while assessing the performances of the various few-shot examples selection approaches for the code explanation task.