Exploring Automated Code Evaluation Systems and Resources for Code Analysis: A Comprehensive Survey

TL;DR

This comprehensive survey examines automated code evaluation systems, their applications, available resources, and machine learning-driven coding tasks, highlighting the Aizu Online Judge platform as a key example.

Contribution

It provides a detailed categorization of AES applications, explores available datasets and resources, and discusses the design, operation, and research aspects of the Aizu Online Judge platform.

Findings

AESs are used in contests, education, recruitment, and online compilers.

Various datasets and resources support research and machine learning tasks.

The Aizu Online Judge platform exemplifies scalable, transparent AES with open data.

Abstract

The automated code evaluation system (AES) is mainly designed to reliably assess user-submitted code. Due to their extensive range of applications and the accumulation of valuable resources, AESs are becoming increasingly popular. Research on the application of AES and their real-world resource exploration for diverse coding tasks is still lacking. In this study, we conducted a comprehensive survey on AESs and their resources. This survey explores the application areas of AESs, available resources, and resource utilization for coding tasks. AESs are categorized into programming contests, programming learning and education, recruitment, online compilers, and additional modules, depending on their application. We explore the available datasets and other resources of these systems for research, analysis, and coding tasks. Moreover, we provide an overview of machine learning-driven coding tasks, such as bug detection, code review, comprehension, refactoring, search, representation, and repair. These tasks are performed using real-life datasets. In addition, we briefly discuss the Aizu Online Judge platform as a real example of an AES from the perspectives of system design (hardware and software), operation (competition and education), and research. This is due to the scalability of the AOJ platform (programming education, competitions, and practice), open internal features (hardware and software), attention from the research community, open source data (e.g., solution codes and submission documents), and transparency. We also analyze the overall performance of this system and the perceived challenges over the years.