Introducing ReQuEST: an Open Platform for Reproducible and Quality-Efficient Systems-ML Tournaments

TL;DR

ReQuEST is an open platform designed to facilitate reproducible, standardized co-design tournaments for optimizing machine learning systems across hardware and software, promoting best practices and benchmarking.

Contribution

It introduces a community-driven, open-source tournament platform leveraging Collective Knowledge for reproducibility and comparison of ML system efficiency and quality.

Findings

Establishes a public repository of optimized ML algorithms and systems.

Provides a standardized framework for reproducible system evaluations.

Facilitates multi-objective co-design competitions for emerging workloads.

Abstract

Co-designing efficient machine learning based systems across the whole hardware/software stack to trade off speed, accuracy, energy and costs is becoming extremely complex and time consuming. Researchers often struggle to evaluate and compare different published works across rapidly evolving software frameworks, heterogeneous hardware platforms, compilers, libraries, algorithms, data sets, models, and environments. We present our community effort to develop an open co-design tournament platform with an online public scoreboard. It will gradually incorporate best research practices while providing a common way for multidisciplinary researchers to optimize and compare the quality vs. efficiency Pareto optimality of various workloads on diverse and complete hardware/software systems. We want to leverage the open-source Collective Knowledge framework and the ACM artifact evaluation methodology to validate and share the complete machine learning system implementations in a standardized, portable, and reproducible fashion. We plan to hold regular multi-objective optimization and co-design tournaments for emerging workloads such as deep learning, starting with ASPLOS'18 (ACM conference on Architectural Support for Programming Languages and Operating Systems - the premier forum for multidisciplinary systems research spanning computer architecture and hardware, programming languages and compilers, operating systems and networking) to build a public repository of the most efficient machine learning algorithms and systems which can be easily customized, reused and built upon.