Evaluation as a Service architecture and crowdsourced problems solving implemented in Optil.io platform

TL;DR

This paper presents the Optil.io platform, an Evaluation as a Service architecture that facilitates reliable, real-time algorithm assessment through crowdsourced challenges and cloud-based evaluation tools.

Contribution

It introduces the Optil.io platform and demonstrates its application in organizing crowdsourced algorithm evaluation challenges.

Findings

Optil.io enables real-time evaluation of algorithms.

The platform supports defining problems and storing evaluation data.

Crowdsourced challenges improve algorithm assessment reliability.

Abstract

Reliable and trustworthy evaluation of algorithms is a challenging process. Firstly, each algorithm has its strengths and weaknesses, and the selection of test instances can significantly influence the assessment process. Secondly, the measured performance of the algorithm highly depends on the test environment architecture, i.e., CPU model, available memory, cache configuration, operating system's kernel, and even compilation flags. Finally, it is often difficult to compare algorithm with software prepared by other researchers. Evaluation as a Service (EaaS) is a cloud computing architecture that tries to make assessment process more reliable by providing online tools and test instances dedicated to the evaluation of algorithms. One of such platforms is Optil.io which gives the possibility to define problems, store evaluation data and evaluate solutions submitted by researchers in almost real time. In this paper, we briefly present this platform together with four challenges that were organized with its support.