CodeLabeller: A Web-based Code Annotation Tool for Java Design Patterns and Summaries

TL;DR

CodeLabeller is a web-based tool designed to efficiently annotate Java source code with design patterns and summaries, facilitating the creation of labeled datasets for supervised learning in software engineering.

Contribution

It introduces an online platform that streamlines code annotation at scale, validated through a large corpus and expert usability evaluation.

Findings

Successfully labeled over a thousand Java files with design patterns and summaries.

Achieved high usability scores indicating ease of use and meeting annotator needs.

Demonstrated practical applicability in education and expert rating of software artifacts.

Abstract

While constructing supervised learning models, we require labelled examples to build a corpus and train a machine learning model. However, most studies have built the labelled dataset manually, which in many occasions is a daunting task. To mitigate this problem, we have built an online tool called CodeLabeller. CodeLabeller is a web-based tool that aims to provide an efficient approach to handling the process of labelling source code files for supervised learning methods at scale by improving the data collection process throughout. CodeLabeller is tested by constructing a corpus of over a thousand source files obtained from a large collection of open source Java projects and labelling each Java source file with their respective design patterns and summaries. Twenty five experts in the field of software engineering participated in a usability evaluation of the tool using the standard User Experience Questionnaire online survey. The survey results demonstrate that the tool achieves the Good standard on hedonic and pragmatic quality standards, is easy to use and meets the needs of the annotating the corpus for supervised classifiers. Apart from assisting researchers in crowdsourcing a labelled dataset, the tool has practical applicability in software engineering education and assists in building expert ratings for software artefacts.