Omnibenchmark: transparent, reproducible, extensible and standardized orchestration of solo and collaborative benchmarks
Izaskun Mallona, Almut Luetge, Ben Carrillo, Daniel Incicau, Reto Gerber, Aidan Meara, Anthony Sonrel, Charlotte Soneson, Mark D. Robinson

TL;DR
Omnibenchmark is a flexible, transparent, and reproducible benchmarking system that enables standardized design, execution, and sharing of performance assessments for data analysis tools and systems, supporting solo and collaborative efforts.
Contribution
It introduces a novel benchmarking framework with flexible syntax, dynamic workflow generation, and reproducible environments, enhancing transparency and extensibility in benchmarking processes.
Findings
Supports forkable and extendable benchmark designs
Provides versioned, standardized output results
Facilitates transparent and reproducible benchmarking workflows
Abstract
Benchmarking involves designing, running and disseminating rigorous performance assessments of methods, most often for data analysis and software tools, but the process can also be applied to experimental systems. Ideally, a benchmarking system is used to facilitate the benchmarking process by providing a structured entrypoint to design, coordinate, execute, and store standardized benchmarks. We describe a novel benchmarking system, Omnibenchmark, that facilitates benchmark formalization and execution in both solo and community efforts. Omnibenchmark provides a flexible benchmark plan syntax (i.e., a configuration YAML file), dynamic workflow generation based on Snakemake, S3-compatible storage handling, and reproducible software environments using environment modules, Apptainer or Conda. Such a setup provides an unprecedented flexibility such that existing benchmark designs can be…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsScientific Computing and Data Management · Bioinformatics and Genomic Networks · Gene expression and cancer classification
