LLM-based Content Classification Approach for GitHub Repositories by the README Files

TL;DR

This paper develops an LLM-based method to automatically classify sections of GitHub README files, significantly improving accuracy and efficiency, and explores parameter-efficient fine-tuning techniques as economical alternatives.

Contribution

It introduces a fine-tuning approach for LLMs to classify README sections, outperforming existing methods and incorporating PEFT techniques like LoRA for cost-effective training.

Findings

Achieved an F1 score of 0.98 in classification accuracy.

Demonstrated the effectiveness of PEFT techniques like LoRA.

Outperformed current state-of-the-art methods.

Abstract

GitHub is the world's most popular platform for storing, sharing, and managing code. Every GitHub repository has a README file associated with it. The README files should contain project-related information as per the recommendations of GitHub to support the usage and improvement of repositories. However, GitHub repository owners sometimes neglected these recommendations. This prevents a GitHub repository from reaching its full potential. This research posits that the comprehensiveness of a GitHub repository's README file significantly influences its adoption and utilization, with a lack of detail potentially hindering its full potential for widespread engagement and impact within the research community. Large Language Models (LLMs) have shown great performance in many text-based tasks including text classification, text generation, text summarization and text translation. In this study, an approach is developed to fine-tune LLMs for automatically classifying different sections of GitHub README files. Three encoder-only LLMs are utilized, including BERT, DistilBERT and RoBERTa. These pre-trained models are then fine-tuned based on a gold-standard dataset consisting of 4226 README file sections. This approach outperforms current state-of-the-art methods and has achieved an overall F1 score of 0.98. Moreover, we have also investigated the use of Parameter-Efficient Fine-Tuning (PEFT) techniques like Low-Rank Adaptation (LoRA) and shown an economical alternative to full fine-tuning without compromising much performance. The results demonstrate the potential of using LLMs in designing an automatic classifier for categorizing the content of GitHub README files. Consequently, this study contributes to the development of automated tools for GitHub repositories to improve their identifications and potential usages.