Towards a Dataset of Programming Contest Plagiarism in Java

TL;DR

This paper introduces the first dataset of Java programming contest solutions for plagiarism detection, enabling evaluation of detection tools and revealing their limitations in contest scenarios.

Contribution

It presents a novel dataset of plagiarized and non-plagiarized Java solutions, with variations to assess detection tools under different conditions.

Findings

Detection tools perform poorly on contest plagiarism cases.

Token-based tools outperform other methods in the dataset.

Template code significantly affects detection accuracy.

Abstract

In this paper, we describe and present the first dataset of source code plagiarism specifically aimed at contest plagiarism. The dataset contains 251 pairs of plagiarized solutions of competitive programming tasks in Java, as well as 660 non-plagiarized ones, however, the described approach can be used to extend the dataset in the future. Importantly, each pair comes in two versions: (a) "raw" and (b) with participants' repeated template code removed, allowing for evaluating tools in different settings. We used the collected dataset to compare the available source code plagiarism detection tools, including state-of-the-art ones, specifically in their ability to detect contest plagiarism. Our results indicate that the tools show significantly worse performance on the contest plagiarism because of the template code and the presence of other misleadingly similar code. Of the tested tools, token-based ones demonstrated the best performance in both variants of the dataset.