Distilled GPT for Source Code Summarization
Chia-Yi Su, Collin McMillan

TL;DR
This paper introduces a small, open-source model trained via knowledge distillation from GPT-3.5 to generate source code summaries, enabling privacy-preserving code documentation without relying on third-party APIs.
Contribution
We develop a 350-million-parameter model trained on GPT-3.5 outputs for source code summarization, offering an open-source alternative that maintains high performance.
Findings
Model mimics GPT-3.5's code summarization capabilities
Runs efficiently on a single 16GB GPU
Provides a privacy-preserving solution for code documentation
Abstract
A code summary is a brief natural language description of source code. Summaries are usually only a single sentence long, and yet form the backbone of developer documentation. A short descriptions such as "changes all visible polygons to the color blue" can give a programmer a high-level idea of what code does without the effort of reading the code itself. Recently, products based on Large Language Models such as ChatGPT have demonstrated a strong ability to write these descriptions automatically. However, to use these tools, programmers must send their code to untrusted third parties for processing (e.g., via an API call). This loss of custody is not acceptable to many organizations. In this paper, we present an alternative: we train an open source model using sample output generated by GPT-3.5 in a process related to knowledge distillation. Our model is small enough (350m parameters)…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsSoftware Engineering Research · Topic Modeling · Machine Learning and Data Classification
MethodsRefunds@Expedia|||How do I get a full refund from Expedia? · Multi-Head Attention · 15 Ways to Contact How can i speak to someone at Delta Airlines · Attention Is All You Need · Cosine Annealing · Weight Decay · Linear Layer · Attention Dropout · Softmax · Dense Connections
