Bencher: Simple and Reproducible Benchmarking for Black-Box Optimization

TL;DR

Bencher is a modular, containerized benchmarking framework for black-box optimization that isolates benchmarks in virtual environments, enabling reproducible, conflict-free evaluation across diverse benchmarks and deployment settings.

Contribution

Bencher introduces a decoupled, containerized benchmarking framework with a unified RPC interface, simplifying integration and reproducibility of diverse black-box optimization benchmarks.

Findings

Supports 80 benchmarks across various domains

Enables deployment on local, HPC, or cloud environments

Reduces dependency conflicts and setup complexity

Abstract

We present Bencher, a modular benchmarking framework for black-box optimization that fundamentally decouples benchmark execution from optimization logic. Unlike prior suites that focus on combining many benchmarks in a single project, Bencher introduces a clean abstraction boundary: each benchmark is isolated in its own virtual Python environment and accessed via a unified, version-agnostic remote procedure call (RPC) interface. This design eliminates dependency conflicts and simplifies the integration of diverse, real-world benchmarks, which often have complex and conflicting software requirements. Bencher can be deployed locally or remotely via Docker or on high-performance computing (HPC) clusters via Singularity, providing a containerized, reproducible runtime for any benchmark. Its lightweight client requires minimal setup and supports drop-in evaluation of 80 benchmarks across continuous, categorical, and binary domains.