Simple and Effective Baselines for Code Summarisation Evaluation
Jade Robinson, Jonathan K. Kummerfeld

TL;DR
This paper proposes a simple LLM-based scoring method for code summaries that considers code context, outperforming traditional metrics and offering a versatile tool for evaluating code documentation quality.
Contribution
Introduces a novel LLM-based scoring baseline for code summarization evaluation that incorporates code context and can operate without reference summaries.
Findings
LLM-based scoring matches or exceeds existing metrics
The method can evaluate documentation quality without references
Combining LLM and embedding methods reduces bias
Abstract
Code documentation is useful, but writing it is time-consuming. Different techniques for generating code summaries have emerged, but comparing them is difficult because human evaluation is expensive and automatic metrics are unreliable. In this paper, we introduce a simple new baseline in which we ask an LLM to give an overall score to a summary. Unlike n-gram and embedding-based baselines, our approach is able to consider the code when giving a score. This allows us to also make a variant that does not consider the reference summary at all, which could be used for other tasks, e.g., to evaluate the quality of documentation in code bases. We find that our method is as good or better than prior metrics, though we recommend using it in conjunction with embedding-based methods to avoid the risk of LLM-specific bias.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsNatural Language Processing Techniques · Topic Modeling · Advanced Text Analysis Techniques
