Online Submission and Evaluation System Design for Competition Operations

TL;DR

This paper introduces an online system that automates submission management and evaluation for research competitions, improving efficiency and compatibility across diverse participant environments.

Contribution

The paper presents a novel online competition platform that streamlines submission handling and evaluation, addressing operational challenges faced by organizers.

Findings

System successfully used in multiple competitions

Automates large-scale submission management

Ensures environment compatibility during evaluation

Abstract

Research communities have developed benchmark datasets across domains to compare the performance of algorithms and techniques However, tracking the progress in these research areas is not easy, as publications appear in different venues at the same time, and many of them claim to represent the state-of-the-art. To address this, research communities often organise periodic competitions to evaluate the performance of various algorithms and techniques, thereby tracking advancements in the field. However, these competitions pose a significant operational burden. The organisers must manage and evaluate a large volume of submissions. Furthermore, participants typically develop their solutions in diverse environments, leading to compatibility issues during the evaluation of their submissions. This paper presents an online competition system that automates the submission and evaluation process for a competition. The competition system allows organisers to manage large numbers of submissions efficiently, utilising isolated environments to evaluate submissions. This system has already been used successfully for several competitions, including the Grid-Based Pathfinding Competition and the League of Robot Runners competition.