Towards Realistic Optimization Benchmarks: A Questionnaire on the Properties of Real-World Problems
Koen van der Blom, Timo M. Deist, Tea Tu\v{s}ar, Mariapia Marchi,, Yusuke Nojima, Akira Oyama, Vanessa Volz, Boris Naujoks

TL;DR
This paper investigates the properties of real-world optimization problems through a questionnaire, aiming to improve the design of benchmarks that better reflect real-world challenges and facilitate more relevant algorithm performance assessments.
Contribution
It introduces a questionnaire to identify key properties of real-world problems and discusses challenges in designing realistic benchmarks based on initial responses.
Findings
Identified key properties of real-world optimization problems.
Highlighted challenges in benchmark design.
Proposed collecting more data for comprehensive analysis.
Abstract
Benchmarks are a useful tool for empirical performance comparisons. However, one of the main shortcomings of existing benchmarks is that it remains largely unclear how they relate to real-world problems. What does an algorithm's performance on a benchmark say about its potential on a specific real-world problem? This work aims to identify properties of real-world problems through a questionnaire on real-world single-, multi-, and many-objective optimization problems. Based on initial responses, a few challenges that have to be considered in the design of realistic benchmarks can already be identified. A key point for future work is to gather more responses to the questionnaire to allow an analysis of common combinations of properties. In turn, such common combinations can then be included in improved benchmark suites. To gather more data, the reader is invited to participate in the…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdvanced Multi-Objective Optimization Algorithms · Metaheuristic Optimization Algorithms Research · Optimal Experimental Design Methods
