Integrating a large-scale testing campaign in the CK framework

TL;DR

This paper explores how the Collective Knowledge framework can facilitate large-scale testing campaigns in programming languages research, demonstrated through an OpenCL compiler testing project, emphasizing ease of use and effective result management.

Contribution

It presents a practical application of the Collective Knowledge framework for large-scale testing, including implementation details and its benefits for managing complex experimental campaigns.

Findings

Effective storage and representation of test cases and results

Simplified management of large testing campaigns

Public availability of the implementation

Abstract

We consider the problem of conducting large experimental campaigns in programming languages research. Most research efforts require a certain level of bookkeeping of results. This is manageable via quick, on-the-fly infrastructure implementations. However, it becomes a problem for large-scale testing initiatives, especially as the needs of the project evolve along the way. We look at how the Collective Knowledge generalized testing framework can help with such a project and its overall applicability and ease of use. The project in question is an OpenCL compiler testing campaign. We investigate how to use the Collective Knowledge framework to lead the experimental campaign, by providing storage and representation of test cases and their results. We also provide an initial implementation, publicly available.